Internship IIAS
General info
- Experience type: Internship
- Start date: Apr 2024
- End date: May 2024
- Role: Data Engineer
My experience at IIAS started during a transition phase from a PostgreSQL-based data warehouse to a ClickHouse-based one. ClickHouse is a much faster database, which is especially useful for large-scale analytics and data cleaning.
During this experience, I had two main tasks:
- Migrate some PostgreSQL pipelines to ClickHouse, using dbt for orchestration.
- Integrate a new application’s data into the new data warehouse.
The second task was the most interesting. It required diving into information stored in the application’s database, which had been accumulating for years. Working with different people, such as doctors and IT staff, was necessary to make sense of the data sometimes. It also required anticipating what data scientists would need in the following year and isolating meaningful data from the rest.
This was my first experience working with big data. Being in a research institute also gave me a better understanding of that environment.