PySpark Developer (Inside IR35)

Standort: London, England Gehalt: Negotiable
Bereich: Consultancy Bereich: Freiberufler
Reference #: CR/080963_1624438379
  • Position :- PySpark Developer
  • Start :-ASAP
  • End Date :-6 Months + possible extension
  • Location :- Right now remote, after pandemic on-site (London, UK)
  • Contract is Inside IR-35

Key responsibilities:

  • Fundamentals of Spark using the Dataframe API
  • Understanding partitioning of data
  • Analysing and performance tuning Spark queries e.g. looking at the DAG
  • Knowledge of Hadoop and its ecosystem of technologies especially Hive Python, OOP concepts using Python
  • Knowledge of Conditional Statements & Loops: If-else Control Structures, For/While Loops
  • Demonstrate a comprehensive understanding of Complex Data Types: Shallow & Deep Copies, Working with Lists & Tuples, Dictionaries & Sets
  • Understand Fundamental Data Structures & their Implementation
  • Good knowledge of Exceptions & Command Line Arguments
  • Contributes to quality assurance by writing unit and functional tests.
  • Ensures development happens for all Software Components in accordance with Detailed Software Requirements specification, the functional design and the technical design document.
  • Basic knowledge of UNIX
  • Demonstrate source control knowledge (preferably GIT)
  • Ability to analyse databases directly using query language tools such as SQL
  • Experience on ETL process on Big Data
  • Have an understanding of data relationships, normalization