Retro on my recent freelancing project

My client faced a challenging task of generating individualized student reports, each spanning 1-2 pages, featuring charts and personalized feedback. The manual process involved creating a separate document for each student, which was time-consuming and inefficient.

She posed the question, "Would it be possible to automate this process?"

This query sparked a web development project. After a month of numerous trials, errors, and several in-person discussions with my client, we successfully launched a simple web application. This new tool streamlined the entire report generation process, significantly reducing the time and effort required to produce these customized documents.

To summarize the process involved in developing the web application:

1. Research: Investigate available solutions to determine the best approach.
2. Prototype Development: Build initial prototypes to explore viable options.
3. Test Run: Conduct initial testing to evaluate the functionality of prototypes.
4. Component Assembly: Build and integrate all necessary components.
5. Styling: Apply CSS and aesthetic elements to enhance the user interface.
6. Usability Testing: Test the application with potential users to ensure it is user-friendly.
7. Documentation: Create tutorial videos and a concise user manual to aid users.
8. Feature Enhancement: Add additional features to improve functionality and user experience.
9. Bug Fixes: Identify and resolve any issues or glitches.
10. Project Review: Write a retrospective to review the project’s successes and learning points (I'm writing this one now).

My initial approach to the project involved using Google Apps Script to integrate Google Spreadsheets and Google Documents. While I was able to handle the plotting aspect sufficiently, I encountered significant challenges with programmatically controlling the positioning of elements within a Google Document. The styling aspect was particularly difficult to manage, as the tools provided did not offer the level of detail and control needed to achieve the desired layout and appearance of the reports.

So I turned to the next alternative, which was Python. The decision proved to be a strategic decision due to the capabilities of the Pandas library paired with Matplotlib, which simplified the plotting tasks immensely. These libraries provided granular control over the elements in a plot, such as checking the height of bars with bar.get_height() and setting the width declaratively. Python's intuitive syntax made the development process smoother.

However, I faced challenges with document styling using the python-docx library. Although it offered many useful methods for document creation, achieving fine-grained stylistic adjustments, like rounded edges on bar charts, was difficult. The need for precise customization to meet specific aesthetic requirements presented a significant hurdle.

Additionally, the project required a frontend interface. This would allow my client to interact with the system, initiating the document creation process from the backend. Designing this user interface involved careful planning to ensure it was both functional and user-friendly, bridging the gap between the Python backend and the client's needs.

Turning to TypeScript, specifically combined with the NextJS framework, TailwindCSS, and the D3 library, marked a significant shift in the project. D3 is a comprehensive data visualization library that offers detailed control and aesthetic options for various chart types, including donut, radar, and multiple bar charts. This made chart generation not only more straightforward but also visually appealing compared to previous attempts.

NextJS provided a robust structure for the web application, integrating several essential layers:

1. Security Layer: Implemented through authentication middleware, ensuring secure access to the application’s functionalities.
2. Presentation Layer: Managed through routing, allowing for structured navigation within the app (e.g., /upload, /report/:id).
3. Data Layer: Facilitated connections to persistent storage, which was crucial for managing the data efficiently.

The application processed an input file, extracting and computing meaningful data from each row of a spreadsheet. This data was then stored in a database, where it could be retrieved either by specific identifiers or through various filters. This setup not only enhanced the functionality and user experience of the app but also provided a scalable solution to handle data manipulation and presentation efficiently.

NextJS streamlined the deployment of the web application, making it straightforward to connect a remote Git repository to Vercel for hosting. This setup enabled quick updates; future iterations involving bug fixes and feature enhancements could be reflected on the web pages in under a minute.

The ongoing interaction with the client during in-person meetings brought forth requests for additional features, such as a printing functionality that preserved layout settings, and adjustments like item ordering and pagination. These requests highlighted that the development process is never truly "finished." As long as there were enhancements to be made or issues to be addressed, the project remained active. Every time I thought I had reached a final version ready for handoff, a new request or suggestion would emerge.

This experience has reminded me of the nature of development. I've learned to anticipate that an app might fail or be initially insufficient in functionalities. Thus, it's crucial that the app is designed to be easy to change. Adopting a mindset that expects and plans for future changes not only makes the development process more manageable but also results in a more robust and adaptable application. Such foresight ensures that the app remains functional and relevant as user needs evolve, making it a far more satisfying project to work on.

There are still features we're looking to add. Some functionalities aren't possible right now due to other dependencies. For example, my client uses FlexiQuiz, which is great but doesn’t allow detailed control over embedded audio files. We could solve this with a new app, but that's outside this project's scope.

Looking back, I truly enjoyed collaborating with my client. I learned a lot from the experience and will take these lessons forward in my future projects.