CATcher:
MarkBind:
RepoSense:
TEAMMATES:
In order to work on CATcher and WATcher, I had to learn how to use Angular. With a background in react, it was a difficult transition due to the added checks and strict nature of Angular.
Below are a few of my learning points:
I learned Angular through various Youtube tutorials, Udemy tutorials, reading the documentation and trying out different things through personal test projects venturing into Angular.
Angular uses TypeScript, so I needed to learn TypeScript. I had only a background in JavaScript while working with React and learning TypeScript had its own difficulties. Below are a few of my learning points:
I learned TypeScript through Youtube tutorials
As an area I have litte experience in, I wanted to dive into the CI/CD pipeline of CATcher and WATcher, gain an understanding of how it works and contribute to make it better. Below are a few of my learning points:
I learned CI/CD through inspecting the code base, trying out different workflows in my own repos and youtube tutorials
Code quality is always important but is especially so when there are so many people working on the same project. Since large portions of WATcher was copied from CATcher, WATcher was made overtly large with a great number of redundant code. It was very poor code quality and the importance of code quality was made clear.
filteredData variable for storing data after it has been filteredI learned about code quality through analysing the responses of seniors to my own pull requests as well as other's pull requests, supplementing my knowledge by reading articles on code quality both generally and specific to web development
Testing is another important part of any project as it reduces the occurrence of major errors and bugs throughout development. With little prior experience in testing, I sought to learn more about it and apply it in WATcher.
describe(string, function) houses related specs labeled by string and defined as functionit(string, function) defines a spec labeled by string and defined as functionexpect(any).toBeFalse defines an expectation of any. There are a large number of matchers for any possible comparisonbeforeEach(function) defines a function to be called before each of the specs in this describe blockcreateSpyObj(string, string[]) creates a spy object that acts as a stub for classes that are depended on by what is being tested. Spies can track calls to it and all argumentsUnderpinning the development of CATcher and WATcher, it was of paramount importance to understand the nuances of the Angular framework. This presented a challenge, transitioning from ReactJs - a framework I was comfortable with. The structure of Angular, contrasted with React's flexibility, necessitated a deep and rigorous engagement with Angular's ecosystem.
*ngIf for conditional rendering, and attribute directives to modify behaviors of DOM elements dynamically. This exploration provided practical insights into complex DOM operations without full page reloads, facilitating rich, responsive user interactions.@angular/forms library, which are a crucial aspect of an application that consists of Form-based components. Angular's form validation is highly robust, integrated deeply with its reactive and template-driven forms, facilitating complex form handling with built-in validators and custom validation functions. In contrast, React forms often require additional libraries for similar levels of validation robustness.Through contributions and an extensive understanding of the codebase, I have attained a certain degree of comfort with Angular as a frontend framework, and will further practice the use of Angular and its features in personal projects.
Incorporating Docker into the WATcher documentation development process was a strategic move to standardize and streamline the development environment. My involvement with setting up a Dev Container using Docker provided valuable insights into containerization and its impact on development workflows.
devcontainer.json), I was able to define and manage configurations such as environment variables, port forwarding, and startup commands efficiently.Working with Docker deepened my understanding of containerization technologies and their role in modern software development. It highlighted the importance of infrastructure as code (IaC) in automating and simplifying development processes. It reinforced best practices in DevOps, particularly in terms of environment standardization.
As part of maintaining development tools, I worked on migrating the project from using TSLint (which is now deprecated), to ESLint. This helped me understand the true role of linters, and how they are defined for a project.
tslint-to-eslint, which attempts to convert configurations, many rules required manual adjustments to align with ESLint’s syntax and capabilities, as well as to not make too many disruptive changes to the codebase.The migration to ESLint has not only streamlined the development environment but also enriched my understanding of effective coding practices.
While developing tests for the ErrorHandlingService and MilestoneService in WATcher, I gained significant insights into Jasmine's powerful features and how they can be leveraged to create thorough, reliable, and maintainable test suites.
MatSnackBar and LoggingService, I could simulate their behavior and assert that they were being called correctly without actually triggering real side effects.GithubService using Jasmine's createSpyObj method, which allowed us to simulate its fetchAllMilestones method without actual HTTP requests. This approach isolates the test environment from external dependencies.handleError() method's conditional logic, which dictates different responses based on the error type, highlighted the importance of comprehensive test paths. I learned to utilize Jasmine's it blocks effectively to specify and isolate each logical branch within the method. This practice ensures that every potential execution path is tested, which is crucial for error handling logic where different types of errors can lead to different user experiences.This exploration into Jasmine's capabilities not only enhanced my technical skills but also deepened my understanding of strategic test planning and execution in a behavior-driven development context. The experience emphasized the value of detailed, well-structured tests in maintaining high software quality and reliability.
A rather inconspicuous but significant learning point, while working on WATcher and CATcher, was UI and UX design. Since the main aim of these applications is to assist students, tutors, professors to understand, contextualise and identify key information with ease, several design decisions had to made from the point of view of how it would most benefit the users.
Some of these included:
#361 Make ItemsPerPage common for all card views
#363 Remodel the design of the Filter bar
#307 Add tool tip for hidden users
#318 Add sorting by Status
#337 Add icon for PRs without milestones
Working on these PRs likely provided a deep dive into the principles of user-centered design, focusing on enhancing the user's journey through intuitive layouts, actionable insights, and consistent behaviors across the application. The challenges often revolved around integrating new features without disrupting the existing user experience, requiring careful consideration of design continuity and user expectations.
Components are the main building blocks for Angular. Each components consists of 3 files:
Refer to the Angular Documentation for guidelines on creating components.
Attribute directives can change the appearance or behavior of DOM elements and Angular components.
For detailed guidance, refer to the Angular Documentation. It provides guidelines on creating and applying attribute directive, covering user events handling and passing values to attribute directive.
In PR #1239, I implemented an attribute directive that listen to click event and will open error snackbar if the target link is an internal link.
NgTemplateOutletNgTemplateOutlet is a directive in Angular that allows for dynamic rendering of templates. It allows the template to be specified along with the context data that should be injected into it.
I utilized NgTemplateOutlet to dynamically render different headers for the card view component based on current grouping criteria. Refer to CardViewComponenet for implementation details.
Jasmine is a behavior-driven development framewrok specific for JavaScript unit testing.
I primarily learned how to use Jasmine from its documentation. I utilized it extensively while working on WATcher test case refactoring. Some relevant PRs include: PR #241, PR #244, PR #245, PR #246, PR #247
describe: Define a group of spec (suite)it: Define a single spec.expect: Create an expectation for a spec.Spy: Mock functions (spies) that can be used to track function calls.When dealing with asynchronous operations like observables, Jasmine provides support through the use of the done function. This allows for effective testing of asynchronous behavior by signaling when a test has completed its execution.
Here's an example from my pull request:
it('should throw error for URL without repo parameter', (done) => {
const urlWithoutRepo = '/issuesViewer';
phaseService.setupFromUrl(urlWithoutRepo).subscribe({
error: (err) => {
expect(err).toEqual(new Error(ErrorMessageService.invalidUrlMessage()));
done(); // Signal that the test has completed
}
});
});
Resources: Angular — Unit Testing recipes (v2+)
It's essential to test for behavior rather than implementation details. This principle was emphasized by a senior in my pull reqeust. By focusing on behavior, tests become more resilient to changes in the codebase and provide better documentation for how components and functions should be used.
Here's an example that illustrates the difference between testing behavior and implementation:
Context: changeRepositoryIfValid will call changeCurrentRepository if repository is valid.
// Test for behavior
it('should set current repository if repository is valid', async () => {
githubServiceSpy.isRepositoryPresent.and.returnValue(of(true));
await phaseService.changeRepositoryIfValid(WATCHER_REPO);
expect(phaseService.currentRepo).toEqual(WATCHER_REPO);
});
// Test for implementation
it('should call changeRepository method if repository is valid', async () => {
githubServiceSpy.isRepositoryPresent.and.returnValue(of(true));
const changeCurrentRepositorySpy = spyOn(phaseService, 'changeCurrentRepository');
await phaseService.changeRepositoryIfValid(WATCHER_REPO);
expect(changeCurrentRepositorySpy).toHaveBeenCalledWith(WATCHER_REPO);
});
The Strategy design pattern allows for the selection of algorithms at runtime by defining a family of interchangeable algorithms and encapsulating each one. It enables flexibility and easy extension of functionality without modifying existing code.
I utilized the Strategy Design Pattern to implement a "Group by" feature that organizes issues / prs based on different criteria.
Implementation of group by feature :
GroupingStrategy): Defines a common interface for all supported grouping strategies.GroupingContextService): This service is used to apply the grouping strategies based on user selection.Of course, Angular is the framework used to run CATcher and WATcher, so learning how it works is an essential part of contributing to the projects. These projects are my first experience using Angular.
As I have experienced React.js and Alpine.js, with experience of working in frontend development during my internship, I expected to pick up Angular with ease. However, slightly different from my expectation, the OOP aspect of Angular makes it quite difficult to pick up.
There are a few interesting concepts that I picked up along the way:
@Component to mark it as an Angular component. This decorator determines a few important properties of the component, including the query selector, the HTML template and the stylesheets.The knowledge of how a component is declared allows me to confidently create a new component in WATcher-PR#235, which was the component to show a list of users with 0 PRs and issues.
One interesting thing about Angular is that it provides a few methods that developers can make use of, to reduce the complexity of component class. This knowledge allows me to make WATcher-PR#230, where I directly modified the Angular model used in the HTML template.
I initially had a lot of trouble trying to understand the operators in RxJS. Ultimately, I was able to understand how it works, and the differences between different operators on an Observable. I was able to see the similarities between different RxJS operators and Java stream methods.
Observable::pipe allows methods to modify the value within the Observable, notably with map and mergeMap.Observable::subscribe listens for changes within the Observable.The knowledge of RxJS operators allow me to modify the underlying processes of the Angular services, and created CATcher-PR#1234, where I set branch for image uploads to main.
One thing to note about RxJS operator is that, Observable pipes are treated as functions, in a sense that they are only called when there is a subscriber. If there are multiple pipes merged into one, each individual pipes are called when there is a subscriber. Consider this example:
Observable a = from(); // some declaration
Observable b = a.pipe(f);
Observable x = b.pipe(g);
Observable y = b.pipe(h);
Observable c = merge(x, y);
c.subscribe();
Notice that c is a merged Observable from pipes of f, g and f, h. So, g and h each are called once, but f is called twice! Imagine if f is a function making multiple API calls to Github.
This knowledge allows me to reduce Github API calls for issues. To get issues from a repository, one must make multiple API calls, each obtaining 100 issues. These API calls are contained within the function f. So instead of splitting the pipe, I refactored to merge g and h and continue the pipe from b:
Observable a = from(); // some declaration
Observable b = a.pipe(f);
Observable c = b.pipe(merge(g, h));
c.subscribe();
Worked on adding Software Project Documentation template to MarkBind, allowing for users to have a starting point for using MarkBind in their project documentation.
Researched into possible integrations of Bun and Bootstrap v5.2 and v5.3 into MarkBind, to determine the value and feasibility of these integrations.
Worked on customizing list icons, such that icons for list items can be customized to apply to the current item only instead of default inheritance to future items.
Worked largely on DevOps side of MarkBind, utilizing GitHub Actions and workflows to handle automation of tasks. These tasks include checking for commit messages in PR descriptions, SEMVER impact labels, reminding adding of new contributors to contributor list.
Researched and implemented the use of DangerJS to automate checking of changes coupling of implementation files to test or documentation files, to ensure that any changes to the repository is properly documented and tested.
Researched into the implementation of automating unassigning of assigned users to issues after a certain period of inactivity.
Researched into common security practices for GitHub Actions, and implementing these practices into the MarkBind repository. These practices are also documented for future contributors to the project.
Learned the underlying workings and how different parts of the codebase are linked together to provide MarkBind's functionalities. From the parser to the renderer, and the different plugins that can be used to extend MarkBind's capabilities. Learned how to implement new features, adding relevant test and documentation to ensure that the codebase is maintainable and modifiable.
Learned how GitHub Actions fits into the development workflow, and how to use it to automate tasks. I used the GitHub Actions documentation to learn about the different types of workflows, how to create and configure workflows, and how to use the different actions available.
Resource: GitHub Actions Documentation
Summary: GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want.
Resource: GitHub Actions Workflow Syntax
Summary: GitHub Actions uses YAML syntax to define the events, jobs, and steps that make up your workflow. You can create a custom workflow or use a starter workflow template to get started.
Resource: GITHUB_TOKEN
Summary: The GITHUB_TOKEN is a GitHub Actions token that is automatically generated by GitHub and used to authenticate in a workflow run. It is scoped to the repository that contains the workflow file, and can be used to perform read and write operations on the repository. It is automatically available to your workflow and does not need to be stored as a secret.
Learned yaml and bash for creation of workflows.
Resource: YAML Syntax
Summary: YAML is a human-readable data serialization standard that can be used in conjunction with all programming languages and is often used to write configuration files. It can also be used in workflows to define the structure of the workflow, including the events, jobs, and steps that make up the workflow.
Resource: Bash Scripting
Summary: Bash is a Unix shell and command language written by Brian Fox for the GNU Project as a free software replacement for the Bourne shell. It has been distributed widely as the shell for the GNU operating system and as a default shell on Linux and OS X.
Resource: Bash Parameter Expansion
Summary: Parameter expansion is a way to manipulate variables in Bash. It is a form of variable substitution that allows for the manipulation of variables in various ways.
Learned how to use other actions in workflows, such as the actions/checkout action to check out a repository onto the runner, allowing subsequent steps to execute operations on the checked-out repository.
Learned how to use DangerJS to aid with workflows.
When to create new workflows (outside of modifiability) Although keeping multiple jobs within the same workflow file is possible, sometimes it may be better not to. Jobs are run based on event triggers such as pull requests etc, but you must add it to the top. This meant that if you had multiple jobs in the same workflow file, they would all run when the event trigger was activated. If you wanted a trigger to only trigger a specific job, you would need to add a check to exclude all other jobs from that trigger.
Pull request trigger by default has the types opened, synchronize, and reopened.
Testing and debugging workflows This can be done locally with the help of Docker and act.
Benefits of local testing: Fast Feedback - Avoid commit/push every time you want to test out the changes.
Resource: Act Usage
Summary: Act reads in your GitHub Actions from .github/workflows/ and determines the set of actions that need to be run. It uses the Docker API to either pull or build the necessary images, as defined in your workflow files and finally determines the execution path based on the dependencies that were defined. Once it has the execution path, it then uses the Docker API to run containers for each action based on the images prepared earlier.
Resource: Docker Docs
Summary:
Steps (PR):
act pull_request -j specific-job -e pr-event.json to run a specific job on the PR event environment.uses:
Can be used to reference an action in the same repository, a public repository, or a published Docker container image. The uses keyword can also be used to reference an action in a private repository by specifying the repository using the repository keyword.
env:
It is best to avoid having expressions ${{ }} in run portion of a step. Instead, env allows defining of variables that store the expression.
awk :
Can be used to extract a section of body, from a line containing START to a line containing END (inclusive of full line).
section=$(echo "$body" | awk '/START/,/END/')
Useful actions:
| Action | Description |
|---|---|
actions/checkout@v3 | Check out a repository onto the runner, allowing subsequent steps to execute operations on the checked-out repository, i.e. gaining access to the repository's code. |
actions/github-script@v7 | Run a script in the GitHub Actions environment using the GitHub CLI. Refer to here |
actions/setup-node@v3 | Set up a Node.js environment for use in workflows. |
actions/stale | Close stale issues and pull requests. |
boundfoxstudios/action-unassign-contributor-after-days-of-inactivity | Automatically unassign user from issues after a certain period of inactivity. |
The definition of inactivity for the GitHub action is any form of history to the issue, be it labeling, comments or references. The action works such that issues and PRs are treated and checked for inactivity separately. This means that any updates done to a PR regarding this issue, will not reset inactivity for the issue.
How unassign and stale actions work:
Stale label to issue or PR based on inactivity (default 60 days)Stale label, then checks whether it's been a set amount of days after the Stale label has been added with no other activity (default 7 days)Stale labelReference workflow of real-life example
Add the Stale label after 6 days and ping a reminder, then have the unassign-contributor-after-days-of-inactivity run 1 day after.
It can also only check on issues that are actually assigned to someone, so that theres no redundancy.
Limitations:
Stale and the user being unassigned despite them actively working on the PR.@username in the reminder due to how the code is written. It is possible to separately ping the user in another comment but that will cause a reset in inactivity. This means slightly lower visibility for the reminder.Building on unassign action, which at some point it might be better off just building our own unassign action for better integration and control
Check corresponding PR (requires more implementation)
Add additional check before setting Stale label to check if corresponding PR has history.
This can be done through checking the list of open PRs and their descriptions whether it mentions the issue. It can also be done through looking at the issue’s history, for PRs that mention it, then checking the history of those PRs. This should be quite manageable since the number of open PRs at any point of time is still relatively low for Markbind’s scale.
Check corresponding issue (requires more implementation) On any activity in PRs, check description to find issues linked to the PR, so activity on PR can be translated to activity on the issue as well by posting a comment on the corresponding issue or something of that nature. This might require checking for a specific issue that has the user assigned to avoid commenting on relevant but not directly linked issues, if the PR has multiple relevant issues. We can also only call it on commits instead of any activity, so as to avoid over-polluting the issue with comments.
Ping after unassign
Same as before, add Stale level after 6 days, but don't need to ping the user, wait until unassign 1 day after, then ping the user that they have been unassigned and if they are still working on it, ask them to reassign themselves. This would likely fit better with a longer timeline.
This solves the visibility problem as it can directly ping the user as resetting the inactivity after the user has been unassigned wouldn't matter.
Implement our own stale action (requires more implementation)
Implement a simplified version of stale action that now allows pinging of user before applying the Stale label.
Due to security reasons, for permissions given to GITHUB_TOKEN, the maximum access for pull requests from public forked repositories is restricted to only read, so it is not possible to add labels since there is no write access. GitHub introduced the pull_request_target event that will bypass this restriction and give the token write access.
Pros:
pull_request events and not pull_request_review events which means need to run on PR merge rather than on PR approved.pull_request event does. This could lead to security vulnerabilities if scripts run are not properly checked for malicious code.
Can be aided by seeking approval before running the job, refer to change repo settingsWorkaround Pros: this can allow for still triggering on PR approved Cons: immensely complicates the workflow
Personal Access Token (PAT)
Create a PAT with the necessary permissions and add it to your repository's secrets. Then, modify the workflow to use this secret instead of the GITHUB_TOKEN.
Pros: this can allow for still triggering on PR approved
Cons: exposes your repository to risks if the forked code can access the token
Specific version tags
When using third-party actions, pin the version with a commit hash rather than a tag to shield your workflow from potential supply-chain compromise, since tags can be adjusted to a malicious codebase state but commit hashes provide immutability.
This can be done by going to the codebase for the specific tag version and looking for the latest commit of the version desired and copying the commit’s full SHA from the url link.
Use:
uses: someperson/post-issue@f054a8b5c1271c37293245628f1cae047eff08c9
Instead of:
uses: someperson/post-issue@v7
Downside is that the updates must be done by updating the commit hash instead of it being done automatically through moving the tag to a new release.
This can be solved by using tools like Dependabot or Renovatebot by adding a comment of the version used, enabling automated dependency updates. Tools like StepSecurity can also be used.
Minimally scoped credentials
Every credential used in the workflow should have the minimum required permissions to execute the job.
In particular, use the ‘permissions’ key to make sure the GITHUB_TOKEN is configured with the least privileges for each job.
permissions can be restricted at the repo, workflow or job level.
Environment variables, like ${{ secrets.GITHUB_TOKEN }}, should be limited by scope, and should be declared at the step level when possible.
Pull_request_target (must be used for write access if PR is from forked repo) Do not use actions/checkout with this as it can give write permission and secrets access to untrusted code. Any building step, script execution, or even action call could be used to compromise the entire repository. This can be fixed by adding code to ensure that the code being checked out belongs to the base branch, which would also be limiting since the code checked out is not up to date for the PR. This can be done using:
- uses: actions/checkout@v4
with:
ref: $
Triggers workflows based on the latest commit of the pull request's base branch.
Even if workflow files are modified or deleted on feature branches, workflows on the default branch aren't affected so you can prevent malicious code from being executed in CI without code review.
Another solution that allows pull_request_target with actions/checkout used on the PR branch, is to add an additional step of running workflow only on approval by trusted users, such that the trusted user has to check the changes in the code from the PR to ensure there is no malicious code before running the workflow.
Untrusted input
Don't directly reference values you do not control, such as echo “${{github.event.pull_request.title}}”, since it can contain malicious code and lead to an injection attack.
Instead use an action with arguments (recommended):
uses: fakeaction/printtitle@v3
with:
title: $
Or bind the value to an intermediate environment variable:
env:
PR_TITLE: $
run: |
echo “$PR_TITLE”
Use OIDC and respective Secrets Manager for access to cloud providers instead of using secrets.
Use GitHub Secrets to store keys securely and reference in workflows using ${{}}.
Can use GitGuardian Shield to help with scanning for security vulnerabilities.
Typescript is a superset of JavaScript that adds static typing to the language. By manipulating variables and functions, Typescript can help catch errors before they occur.
| Syntax | Name | Feature |
|---|---|---|
? | Optional chaining operator | variable returns undefined if doesn't exist. Also used for optional function parameters or class properties |
?? | Nullish coalescing operator | returns the right-hand operand when the left-hand operand is null or undefined. |
! | Non-null assertion operator | variable is not null or undefined, only used if you are sure that value will exist. |
MarkBind uses a monorepo structure, which means that multiple packages are contained in a single repository. The process of upgrading dependencies and packages in MarkBind involves the following steps:
Checking current versions: Check the current versions of the dependencies and packages in the project. This can be done by looking at the package.json file for each project. The command npm ls package_name will output which packages are using what versions.
Review changelog and documentation: Review the changelog and documentation for the dependencies and packages to see what changes have been made in the new versions.
Upgrade dependencies and packages: Update the relevant package.json file or the root one for dependencies across all packages, then run npm run setup
A default package manager for Node.js.
npm install, npm update, npm run <scripts> etc. and how they helped streamline the development process."scripts", "dependencies" and how to manage them..npmignoreA CSS linter that helps enforce conventions and avoid errors.
stylelintrc.js file, a configuration object for Stylelint to suit our own needs.A JavaScript library that provides a framework for building command-line interfaces (CLIs) in Node.js applications
A CI/CD platform allowing developers to automate workflows directly within their GitHub repository.
.yml files in .github/workflow.A set of web developer tools built directly into the Chrome browser.
Network section - disable cache and change network settingsPerformance insights sectionA template engine for Javascript. It provides a way to mix static content with dynamic data.
A Javascript testing framework that focuses on simplicity when writing tests.
jest.mock, jest.fn to implement mocks and jest.spyOn to create spies.A website that documents web technologies for developers. The articles are written by developers that covers a lot of aspects related to the web.
<img> and <script> behaves, along with some common issues eg. lazy loadingWhile working with Vue components this semester, I've learned more about props and script in vue when working on the template for panels through adding a new prop isSeamless and writing new script for the panel component.
MarkBind uses Jest together with Vue Test Utils for its snapshot tests, which test Vue components against their expected snapshots. While updating the component, I wrote new tests to ensure that the Vue components are working as expected.
An interesting issue I've encountered this semester while researching on integrating a full search functionality is the issue of importing esm like pagefind into cjs modules. CommonJS uses the require('something') syntax for importing other modules and ESM uses the import {stuff} from './somewhere' syntax for importing.
Another crucial difference is that CJS imports are synchronous while ESM imports are asynchronous. As such, when importing ES modules into CJS, the normal require('pagefind') syntax would result in an error. Instead, you'll need to use await import('pagefind') to asynchronously import the module. This difference in imports is something that should be taken note of since we use both the ESM import syntax and CJS require syntax in various files in MarkBind.
Nunjucks is a rich and powerful templating language for JavaScript. MarkBind supports Nunjucks for templating and I’ve used Nunjucks specifically to create a set of mappings of topics to their pages, and to write macros.
macro Nunjucks macro allows one to define reusable chunks of content. A great benefit of macro is the reduction of code duplication due to its ability to encapsulate chunks of code into templates and its ability to accept parameters so that the output can be customised based on the inputs provided.
set and import While combining the syntax pages in this commit, I worked on a set that keeps track of the various syntax topics and their information. This was a good exercise to experience how to create a variable using set and import it in other files to access its values using import.
MarkBind has Vue.js components built on the popular BootStrap framework. Much of Bootstrap's features are supported in and out of these components as well. While creating the portfolio template, I got to learn more about the various components and layouts of Bootstrap.
grid Bootstrap grid built with flexbox and is fully responsive. More specific aspects I've learned
Explored various components offered by Bootstrap, such as accordions, cards, carousels
To sync a forked repository with the original repository after discarding all the changes in the forked repository:
git remote add upstream <URL_of_original_repository>
git fetch upstream
git checkout master
git reset --hard upstream/master
git push origin master --force
To append a new commit onto the already existing commit you can do the following:
git add .
git commit --amend
git push origin <branch_name> --force
If there is significant changes to file after renaming, git treat it as a new file and the history of the file is lost. So to preserve the history of the file, need to separate the renaming and the changes (and do rebase and merge).
Use squash merge to keep PR history concise.
r.patch, r.minor, r.major
Use !! To change to boolean.
Use type && {key: value} to quick define type.
Use as to cast type.
use ! to assert non-null.
In functional programming, many methods are bundled in the style like Array.{method}.
Redux/Saga is a predictable state container for JavaScript apps. It is a state management tool, and is often used with React (not very relevant to markbind).Pinia often used for vue.
Vscode's "goto references" does not work well with javascript (mixed inside the typescript). As currently, some of the core packages are not migrated to typescript yet, the references are not recognized. So need to use "findin file" instead.
Possible to auto re-compiling the typescript file into javascript files when it is changed, and only recompile the changed files.
If given not enough parameters, javascript treat missing ones as undefined; if given extra paremeters, javascript will ignore;
Gain knowledge on vue component basics (eg:V-model, v-for, v-once, meaning of <Script setup lang=“ts”> , Reactive variablem ref VS shallowRef (triggerRef) and customRef, How to ref a div without get element by id...)
Gain knowledge on and how new line in vue could be treated as brace and leads to subtle bugs.
Gain knowledge on vue component life cycle knowledge and how "onMounted" can fix the mermaid plugin issue.
Introduced to the concept server side rendering, gain knowledge of the Vue hydration issue.
Workflow vice, a good practice is to not immediately merge a pull request after it is reviewed. Instead, wait for a day or two to see if there are any other comments or suggestions.
For command line tool (like markbind), a good project structure is to have a cli and a core folder. The cli folder contains the code for the command line interface, while the core folder contains the core logic of the project. When building from source, need to npm link the cli folder.
More comfortable with using loggers to debug.
Jest as the testing framework (and debugger).
Jest can mock other tools (fs for example).
Jest can spyon other methods, so track is usage(eg: how many times is it called, what are the arguements the methods are called with)
Snapshot (Recursively comparing every folder and file in the expcted folder with the actual generated files) as a way to do the functional testing.
Cheerio to convert html string to dom, and locate elements in the dom.
Understand the concept of hoisting in JavaScript.
Npm is different from yarn in that it has a flat dependency tree, while yarn has a nested dependency tree. So yarn allows the reuse of the same package in the dependency tree, while npm does not.
Can use npm run to list all the runnable scripts.
Fix issues and simple bugs is the best way to gain familiarity with the codebase.
Understand the difference between inline markdown and non-inline markdown.
Can use comment like eslint-disable-next-line no-await-in-loop to disable eslint for the next line for a specific rule.
Better understand the workflow for frontend: markbind serve -d or npm run build:web to test frontend changes, the frontend markbind.min.js and markbind.min.css bundles are only updated during release.
Gain knowledge on debugging frontend with browser, and understand css inheritance.
The rendering flow for creating a complete website from Markdown using MarkBind involves several key components and processes:
Site instance in Site/index.ts.Site constructor sets up the necessary paths, templates, and initializes various managers and processors.readSiteConfig method in Site/index.ts reads the site configuration file (site.json) using the SiteConfig class.collectAddressablePages method in Site/index.ts collects the addressable pages based on the site configuration.pages and pagesExclude options to determine the valid pages.PluginManager class in plugins/PluginManager.ts handles the initialization and management of plugins.Plugin class.buildAssets method in Site/index.ts builds the necessary assets for the site.generatePages method in Site/index.ts initiates the rendering process for each page.Page instance for each page using the createPage method.Page class in Page/index.ts handles the rendering of individual pages.process method of the NodeProcessor class to process the Markdown content.PluginManager executes the relevant hooks for each plugin.beforeSiteGenerate, processNode, postRender, etc., to modify the page content or perform additional tasks.LayoutManager class in Layout/LayoutManager.ts handles the generation and combination of layouts with pages.Layout class to process and render the layout files.VariableProcessor to replace variables with their corresponding values.MarkBind integrates Vue.js for building user interfaces and enhancing the rendering process. Here's how Vue.js is used in MarkBind:
Server-Side Rendering (SSR): MarkBind utilizes Vue's server-side rendering capabilities to pre-render the pages on the server. The pageVueServerRenderer module in Page/PageVueServerRenderer.ts handles the compilation of Vue pages and the creation of render functions.
Client-Side Hydration: After the initial server-side rendering, the rendered HTML is sent to the client. On the client-side, Vue takes over and hydrates the pre-rendered HTML, making it interactive and reactive.
Vue Components: MarkBind defines various Vue components to encapsulate reusable UI elements and functionality. These components are used throughout the rendered pages to enhance the user experience.
Integration with MarkBind Plugins: MarkBind plugins can also utilize Vue.js to extend the functionality of the application. Plugins can define custom Vue components, directives, and hooks to interact with the rendering process.
MarkBind follows a build process to generate the final website and manage the necessary assets:
Asset Building: The buildAssets method in Site/index.ts handles the building of assets for the site. It copies the specified assets from the site's root path to the output path.
Core Web Assets: MarkBind relies on core web assets such as CSS and JavaScript files. The copyCoreWebAsset method in Site/index.ts copies these assets from the @markbind/core-web package to the output directory.
Plugin Assets: Plugins can also provide their own assets, such as CSS and JavaScript files. The PluginManager handles the collection and copying of plugin assets to the output directory.
Icon Assets: MarkBind supports various icon libraries, including Font Awesome, Glyphicons, Octicons, and Material Icons. The copyFontAwesomeAsset, copyOcticonsAsset, and copyMaterialIconsAsset methods in Site/index.ts handle the copying of these icon assets to the output directory.
Bootstrap Theme: MarkBind allows customization of the Bootstrap theme used in the site. The copyBootstrapTheme method in Site/index.ts handles the copying of the selected Bootstrap theme to the output directory.
MarkBind uses markdown-it for rendering html from markdown files. markdown-it is a fast markdown parser and has very extensive plugins support and great extensibility.
markdown-it through adding a rule to markdown-it's attributeAdding custom rules to markdown-it can be done easily by adding a rule to the attribute.
For example, if we want to add our rules for rendering fenced code blocks, we can do so by adding a rule to the markdown-it's attribute.
markdownIt.renderer.rules.fence = (tokens: Token[],
idx: number, options: Options, env: any, slf: Renderer) => {}
Parameters
token.type: The type of the token (e.g., 'fence', 'code', 'paragraph').token.info: Contains the language specified after the opening set of backticks, if any, plus additional options.token.content: The text content within the fenced code block.fence token within the tokens array. This lets us find tokens before and after the fence if needed.Purpose of the fence renderer rule
The markdownIt.renderer.rules.fence function is responsible for taking a fence token (representing a fenced code block) and converting it into the appropriate HTML output. This could include syntax highlighting, if our setup supports it.
How it Works
Inside the function, we have access to all the information in the tokens, options, and the environment. We can craft custom logic to generate the desired HTML structure for our fenced code blocks. Here's a very basic example:
markdownIt.renderer.rules.fence = (tokens, idx, options, env, slf) => {
const token = tokens[idx];
const content = token.content;
const language = token.info.trim(); // Language after the opening backticks
return `<pre><code class="language-${language}">${content}</code></pre>`;
};
MarkBind uses Cheerio for parsing and manipulating the HTML structure of Markdown files after they have been processed by markdown-it. Cheerio is a fast, flexible, and lean implementation of core jQuery designed specifically for the server.
To use Cheerio, we first need to load HTML into it. This is done by passing the HTML string to the cheerio.load function.
const $ = cheerio.load('<h2 class="title">Hello world</h2>');
The $ variable now contains a Cheerio instance that wraps the parsed HTML, and can be used similarly to how we would use jQuery in the browser.
Cheerio uses CSS selectors to select elements, just like jQuery. Here are some examples:
// Select all h2 elements
$('h2');
// Select the element with id "main"
$('#main');
// Select all elements with class "text"
$('.text');
// Select all a tags within h2 elements
$('h2 a');
Once we have selected elements, we can manipulate them in various ways. Some common methods include:
addClass(className): Adds the specified class to the selected elements.removeClass(className): Removes the specified class from the selected elements.attr(attributeName, value): Gets or sets the value of the specified attribute.text(newText): Gets or sets the text content of the selected elements.html(newHtml): Gets or sets the inner HTML of the selected elements.append(content): Appends the specified content to the end of each selected element.prepend(content): Prepends the specified content to the beginning of each selected element.remove(): Removes the selected elements from the DOM.Here's an example that demonstrates some of these methods:
// Add the class "highlight" to all h2 elements
$('h2').addClass('highlight');
// Set the text of the element with id "title"
$('#title').text('New Title');
// Append a span to each paragraph
$('p').append('<span>Some appended text</span>');
After manipulating the parsed HTML with Cheerio, we can render it back to an HTML string using the html method.
$.html();
//=> '<h2 class="title highlight">New Title</h2><p>Some text<span>Some appended text</span></p>'
This is useful when we need to save the manipulated HTML back to a file or send it as a response in a web application.
Cheerio provides a simple and efficient way to parse and manipulate HTML structures in MarkBind plugins, enabling powerful transformations of the rendered Markdown content.
Vue.js is a progressive JavaScript framework for building user interfaces. It provides a declarative and component-based approach to UI development, making it easier to create and maintain complex applications.
Vue allows we to extend the behavior of HTML elements or Vue components through custom directives. Custom directives provide a way to encapsulate and reuse DOM manipulation logic across your application.
Vue.directive('my-directive', {
bind(el, binding, vnode) {
// Directive initialization logic
},
inserted(el, binding, vnode) {
// Logic to be executed when the directive is inserted into the DOM
},
update(el, binding, vnode, oldVnode) {
// Logic to be executed when the directive's bound value changes
},
componentUpdated(el, binding, vnode, oldVnode) {
// Logic to be executed after the containing component's VNode has updated
},
unbind(el, binding, vnode) {
// Cleanup logic when the directive is unbound from the element
}
})
Parameters
binding.value: The value passed to the directive. It can be a primitive value, an object, or a function.binding.oldValue: The previous value of the directive, only available in the update and componentUpdated hooks.binding.arg: The argument passed to the directive, if any, denoted by a colon (e.g., v-my-directive:arg).binding.modifiers: An object containing any modifiers applied to the directive (e.g., v-my-directive.modifier).update and componentUpdated hooks.Directive Lifecycle Hooks
Custom directives have access to several lifecycle hooks that allow we to execute logic at different stages:
Usage
To use a custom directive, we can attach it to an element or component using the v- prefix followed by the directive name. For example:
<div v-my-directive="value"></div>
In this case, my-directive is the name of the custom directive, and value is the value being passed to the directive.
Custom directives provide a powerful way to encapsulate and reuse DOM manipulation logic in Vue applications. They allow we to extend the behavior of elements and solve specific problems related to integrating external libraries or custom functionality.
When integrating third-party libraries into Vue components, we may encounter scenarios where the library's initialization script doesn't work as expected within the component's lifecycle. This can happen due to timing differences between the library's initialization and the component's rendering process.
One common issue is when a library relies on the presence of certain DOM elements during its initialization, but those elements are dynamically rendered by the Vue component and may not be available when the library's initialization script runs.
To solve this problem, we can leverage Vue's custom directives. By creating a custom directive that handles the library's initialization logic, we can ensure that the initialization happens at the appropriate time within the component's lifecycle.
Here's an example of how we can create a custom directive to initialize a third-party library on a specific element:
Vue.directive('my-library', {
inserted(el) {
// Initialize the library on the element
myLibrary.init(el);
},
unbind(el) {
// Cleanup the library when the directive is unbound
myLibrary.destroy(el);
}
})
In this example, the my-library directive is responsible for initializing the myLibrary on the bound element when it is inserted into the DOM. It also handles the cleanup process when the directive is unbound from the element.
To use this directive in a Vue component, we can simply attach it to the desired element:
<template>
<div v-my-library>
<!-- Element content -->
</div>
</template>
By using the custom directive, we ensure that the library initialization happens at the appropriate time within the component's lifecycle, solving the issue of initialization timing.
CommonJS is a module system used in Node.js and other JavaScript environments. It uses the require() function to import modules and module.exports or exports to export modules.
// Importing a module
const myModule = require('./myModule')
// Exporting a module
module.exports = {
// Exported properties and methods
}
ECMAScript Modules (ESM) is the standard module system introduced in JavaScript with ES6 (ECMAScript 2015). It uses the import and export keywords for importing and exporting modules.
// Importing a module
import myModule from './myModule'
// Exporting a module
export default {
// Exported properties and methods
}
ESM provides benefits such as static analysis, tree shaking, and better performance compared to CJS. However, CJS is still widely used, especially in older codebases and libraries.
require() and module.exports, while ESM uses import and export.Node.js supports both CJS and ESM, and the module system used depends on the file extension (.mjs for ESM and .js for CJS) or the "type": "module" field in the package.json file.
When a web page is loaded, the browser follows a specific order to render the content:
Content partially generated by GenAI
Java is used extensively in the backend for RepoSense, from the generation of the RepoSense report, to the different git commands required to clone repositories and analyse them. It was not difficult to pick up Java as I had some prior experience in Java in previous classes such as CS2030S, CS2040S and CS2103T, but the intricacies surrounding the different Java libraries was something that I was never properly exposed to and had to learn over time as I worked on the project.
Some of the aspects I have learnt regarding Java:
Pattern objects for repeated use insteadDocker is something that I have always wanted to work with, especially so in combination with other container orchestration tools such as Kubernetes. I looked into the possibility of Docker being used to containerise RepoSense, enabling us to better test RepoSense, provide end users with a premade container with RepoSense's dependencies resolved for them, and a way to quickly deploy RepoSense to their favourite cloud providers (e.g. AWS ECS, etc.) for greater availability of RepoSense for their target users.
Some of the aspects I have learnt:
Vue is a frontend JavaScript framework to build web user interfaces, while Pug is a preprocessor that speeds up writing HTML. In RepoSense, Vue brings the interactivity in the generated reports and Pug makes it easier to write more concise and readable HTML.
Having some experience in frontend libraries/frameworks like React, I did not have too steep a learning curve when learning Vue; however, I still took some time to get used to concepts and the syntax of Vue including state management, the idea of single-file components, conditional rendering, reactive UI elements, and much more. One particular aspect I enjoyed learning and implementing in Vue was the ease of declaring state in a component just within the data() function. This was, to me, contrasted with React where useState and useEffect are more complicated and tricky to use.
Cypress is a testing framework that allows for end-to-end testing of web applications. I used it in RepoSense to write tests for the UI.
Cypress was a new tool to me and I had to learn how to write tests using this tool as well as how to set up the test environment. Many Cypress commands are based on natural words like .then, .get, .to.deep, just to name a few, but the concepts of Cypress like asynchonicity, closures, and its inclusion of jQuery make it unfamiliar to me.
...
Pug, formerly known as Jade, is a templating language for Node.js and browsers. It simplifies HTML markup by using indentation-based syntax and offers features like variables, includes, mixins, and conditionals, making web development more efficient and readable.
I learnt how to create a Pug template and integrate it into a Vue component.
StackOverflow, ChatGPT, existing codebase, Pug Website
Vue.js is a progressive JavaScript framework used for building user interfaces and single-page applications. It offers a flexible and approachable structure for front-end development, with features like data binding, component-based architecture, and a simple yet powerful syntax.
I learnt the rationale behind the Single File Component structure, as well as how to implement it to refactor code. It was very similar to React in that the framework is structured around components, and props are still used for data flow. I also learnt how to access local files from within the framework to dynamically load data.
StackOverflow, ChatGPT, existing codebase, Vue Website
Cypress is an end-to-end testing framework used primarily for testing web applications. It provides a comprehensive set of tools and features to automate testing workflows, including real-time testing, automatic waiting, and built-in support for modern JavaScript frameworks.
I learnt how to write simple tests with the framework, as well as how to use the E2E live view to debug and design tests.
StackOverflow, ChatGPT, existing codebase, Cypress Documentation
Markdown-it is a popular JavaScript library used for parsing Markdown syntax and converting it into HTML. It provides a simple and flexible way to format text with lightweight markup syntax.
I learnt how to integrate markdown-it into a Vue component to allow for dynamic parsing of Markdown code into HTML for display.
StackOverflow, ChatGPT, markdown-it Documentation, this guide
Vite is a build tool that offers fast development and optimized production builds for web projects, particularly for Vue.js applications.
I learnt how to configure a Vite project.
StackOverflow, ChatGPT, Vite Documentation, Vue CLI to Vite migration guide
ESLint is a pluggable linting utility for JavaScript and TypeScript that helps identify and fix common coding errors and enforce consistent code style. Stylelint is a linter for CSS and Sass that helps enforce consistent coding conventions and identifies potential errors in stylesheets.
I learnt how to implement ESlint and Stylelint rules and modify the config files.
StackOverflow, ChatGPT, Documentation
Data migration is critical aspect of software development and system maintenance, it involves moving data efficiently while maintaining data integrity, security, and consistency. Having the chance to be involve in data migration really opened my eyes to its general procedure. We were tasked with migrating NoSQL datastore entity to SQL postgresql.
E2E tests are a type of software testing that evaluates the entire workflow of an application from start to finish, simulating real user interactions. The purpose of E2E testing is to ensure that all components of an application, including the user interface, backend services, databases, and external integrations, work together correctly to achieve the desired functionality. Here's an explanation of E2E tests and how they are conducted. As E2E tests are very expensive to run, it is crucial that we identify the important workflows and simulate the actions involved by interacting with the UI. You then assert the expected conditions are present after the interaction. Teammates uses Selenium to locate and interact with the elements in the UI. I have to admit, this is my first time doing tests for Frontend much less the whole application. It was cool to see the browser jump around and simulate the required action. I also saw the value in this as I managed to uncover many bugs that was not caught in earlier tests.
References:
Mockito facilitates unit testing by mocking dependencies. Mock objects are used to simulated objects that mimic the behaviors of real objects in a controlled way, allowing developers to isolate and test specific components of their code without relying on actual dependencies or external systems. While I have written Stubs in CS2103T, this is my first time using a dedicated mocking library and it has changed my life. I also have used what I have learnt in many job interviews.
mock method to initialise the mock objectwhen/then for you to inject the controlled outcomeverify mainly to check number of invocationsReferences:
TEAMMATES uses Hibernate, an Object-Relational Mapper(ORM). ORM are widely used in software development today as it provides several benefit to developers. While I have used ORMs before, such as Prisma, it is my first time using Hibernate. ORMs simplifies database interactions by allowing developers to work with Java objects directly, abstracting away the complexities of SQL queries. Also, as the name suggest, it allows us to map Java Objects to database table and their relationship. Allowing for easier and seamless operations with the database table. I read up on some Hibernate basics:
References:
I was required to deploy a staging environment for the course entity migration. It was my first time using GCP so I managed to gain familiarity with the vast tools that GCP offers. The guides provided by the seniors was just very descriptive and encouraged me to explore tweaking settings to better fit my use case.
References:
As part of the v9-migration, I had to familiarise myself with the Hibernate ORM. It is my first time using Hibernate ORM, as I was only familiar with the Eloquent ORM from Laravel, as well as the ORM from Django. ORMs are extremely beneficial as they essentially translate between data representations used in the application and those in the database. It also makes your code more readable as it simplifies complex queries and makes transitioning between various database engines seamless, should the need arise.
Aspects Learnt:
persist and merge to insert or update an entity respectivelyResources
TEAMMATES uses Solr for full-text search, and is structured to function for both the datastore and SQL databases.
Aspects Learnt:
Resources:
Having only used SQLite and MySQL in the past, I had to familiarise myself with PostgreSQL as it is the SQL database used in TEAMMATES.
Aspects Learnt:
Resources:
Having had no experience utilising Angular prior to working on TEAMMATES, I was introduced to several neat features that Angular has to offer.
Aspects Learnt:
Angular's component-based architecture makes it easy to build and maintain large applications. Each component is encapsulated with its own functionality and is responsible for a specific UI element. This modularity allowed me to quickly understand and contribute to the project, as I could focus on individual components without being overwhelmed by the entire codebase.
Angular's dependency injection system is a design pattern in which a class receives its dependencies from external sources rather than creating them itself. This approach simplifies the development of large applications by making it easier to manage and test components.
Angular offers the trackBy function, which I used in conjunction with the *ngFor directive to manage lists more efficiently. Normally, *ngFor can be inefficient because it re-renders the entire list when the data changes. However, by implementing trackBy, Angular can track each item's identity and only re-render items that have actually changed. This reduces the performance cost, especially in large lists where only a few items change.
When deploying the staging environment for the ARF upgrade, I managed to work with and gain familiarity with the deployment workflow, as well as several GCP tools and the gcloud sdk.
Aspects Learnt
Resources:
Snapshot testing with Jest is an effective strategy to ensure that user interfaces remain consistent despite code changes. It's important for developers to maintain updated snapshots and commit these changes as part of their regular development process.
Snapshot tests are particularly useful for detecting unexpected changes in the UI. By capturing the "snapshot" of an output, developers can compare the current component render against a stored version. If changes occur that aren't captured in a new snapshot, the test will fail, signaling the need for a review.
Mockito is a popular Java-based framework used primarily for unit testing. It allows developers to isolate the units of code they are testing, to focus solely on the component of software that is being tested.
Mockito allows developers to create mock implementations of dependencies for a particular class. This way, developers can isolate the behavior of the class itself without needing the actual dependencies to be active. By using mock objects instead of real ones, tests can be simplified as they don’t have to cater to the complexities of actual dependencies, such as database connections or external services. Mockito also provides tools to verify that certain behaviors happened during the test. For example, it can verify that a method was called with certain parameters or a certain number of times.
Resources:
E2E Testing allows us to ensure that the application functions as expected from the perspective of the user. This type of testing simulates real user scenarios to validate the complete functionality of the application. Common tools for conducting E2E testing include Selenium, Playwright, and Cypress.
Throughout the semester, I had to migrate several E2E tests and also create some new ones as part of the ARF project, which exposed me to the Page Object Model, which allows for easier testing and maintenance. It enhances code reusability as the same Page Object Model can be reused across related test cases.
E2E Tests may be the most complicated type of test to write, as it involves multiple components of the application; testing it as a whole, rather than in isolated components. As such, pinpointing the sources of errors or failures can be difficult. E2E Tests can also be flaky at times, passing in one run, and failing on others, which could occur due to numerous reasons such as timing issues, concurrency problems or subtle bugs that occur under specific circumstances. However, it is still highly useful as it helps to identify issues in the interaction between integrated components, and also simulates real user scenarios.
Resources:
Datastore is a NoSQL document database. While it provides scalability and performance advantages, it falls short when dealing with complex queries. While writing migration scripts, I read up on the following from the Datastore documentation:
References:
TEAMMATES uses Hibernate, an Object-Relational Mapping framework which allows us to interact with the database without writing SQL commands. It abstracts these low-level database interactions, enabling developers to work with high-level objects and queries instead. I read up on some Hibernate basics:
References:
Mockito facilitates unit testing by reducing the setup needed to create and define behaviour of mocked objects. The provided mock, when/then and verify methods not only simplify the test writing process, but also enhance their readability and clarity for future developers.
References:
I was introduced to Docker during the onboarding process. I learnt about containers and the benefits of containerization, such as portability and isolation, and how they enable developers on different infrastructure to work in a consistent environment.
References:
As part of the V9 migration, I had to rewrite the logic to query from the SQL database using Hibernate ORM API instead of from Datastore.
TEAMMATES' back-end code follows the Object-Oriented (OO) paradigm. The code works with objects. This allows easy mapping of objects in the problem domain (e.g. app user) to objects in code (e.g. User).
For the data to persist beyond a single session, it must be stored/persisted into a database. V9 uses PostgreSQL, a relational database management system (RDBMS) to store data.
It is difficult to translate data from a relational model to an object models, resulting in the object-relational impedance mismatch.
A Object/Relational Mapping (ORM) framework helps bridge the object-relational impedance mismatch, allowing us to work with data from an RDBMS in a familiar OO fashion.
Jakarta Persistence, formerly known as Java Persistence API (JPA) is an API for persistence and ORM. Hibernate implements this specification.
The Criteria API allows us to make database queries using objects in code rather than using query strings. The queries can be built based on certain criteria (e.g. matching field).
Using Join<X, Y>, we can navigate to related entities in a query, allowing us to access fields of a related class. For example, when querying a User, we can access its associated Account.
Hibernate maintains a persistence context, which serves as a cache of objects. This context allows for in-code objects to be synced with the data in the database.
Using persist(), merge(), and remove(), we can create, update, and remove an object's data from the database. These methods schedule SQL statements according to the current state of the Java object.
clear() clears the cached state and stops syncing existing Java objects with their corresponding database data. flush() synchronises the cached state with the database state. When writing integration tests, I found it helpful to clear() and flush() the persistence contexts before every test, to ensure that operations done in one test do not affect the others in unexpected ways.
To isolate units in unit testing, it is useful to create mocks or stubs of other components that are used by the unit.
We can create a mock of a class using mock(). We can then use this mocked object as we would a normal object (e.g. calling methods). Afterwards, we can verify several things, such as whether a particular method was called with particular parameters, and how many times a particular method call was performed.
If a method needs to return a value when called, the return value can be stubbed before the method of the mocked object is called. The same method can be stubbed with different outputs for different parameters. Exceptions can also be stubbed in a similar way.
As part of the instructor account request form (ARF) project, I had to create an Angular form.
Angular has 2 form types: template-driven, and reactive.
Template-driven forms have implicit data models which are determined by the form view itself. Template-driven forms are simpler to add, but are more complicated to test and scale.
Reactive forms require an explicitly-defined data model that is then bound to the view. The explicit definition of the model in reactive forms makes it easier to scale and test, particularly for more complex forms.
Standard HTML attributes may still need to be set on Angular form inputs to ensure accessibility. For instance, Angular's Required validator does not set the required attribute on the element, which is used by screen readers, so we need to set it also. Another example would be setting the aria-invalid attribute when validation fails.
To make inline validation messages accessible, use aria-describedby to make it clear which input the error is associated with.
Angular has some built-in validator functions that can be used to validate form inputs, and allows for custom validators to be created. Validators can be synchronous or asynchronous.
By default, all validators run when the input values change. When there are many validators, the form may lag if validation is done this frequently. To improve performance, the form or input's updateOn option can be set to submit or blur to only run the validators on submit or blur.
git rebase can be used to keep branch commit history readable and remove clutter from frequent merge commits.
In particular, the --onto option allows the root to be changed, which is useful when rebasing onto a branch that has itself been modified or rebased.
Each Git commit has a committer date and an author date. When rebasing, the committer date is altered. To prevent this, use --committer-date-is-author-date.
EntityManagers do not always immediate execute the underly SQL statement. One such example is when we create and persist a new entity, the createdAt timestamp is not updated in the entity object in our application until we call flush().
This is because by calling flush() we can ensure that all outstanding SQL statements are executed and that the persistence context and the db is synchronized.
Persistent entities are entities that are known by the persistence provider, Hibernate in this case. An entity(object) can be made persistent by either saving or reading an object from a session. Any changes (e.g., calling a setter) made to persistent entities are automatically persisted into the database.
We can stop hibernate from tracking and automatically updating the entities by calling detach(Entity) or evict(Entity). This will result in the entity becoming detached. While detached, Hibernate will have no longer track the changes made to the entity. To save the changes to the database or make the entity persistent again, we can use merge(Entity).
While using the new SQL db, we often find ourselves needing to refer to another related entity for example FeedbackSessionLogs.setStudent(studentEntity). This would often require us to query the db for the object and then call the setter. This is inefficient especially if we already have information like the studentEntity's primary key.
Hibernate provides a getReference() method which returns a proxy to an entity, that only contains a primary key, and other information are lazily fetched. While creating the proxy, Hibernate does not query the db. Here is an article that goes through different scenarios using reference to see which operations would result in Hibernate performing a SELECT query and which does not. It also includes some information on cached entities in Hibernate.
It is important to note that, since Hibernate does not check that the entity actually exists in the db on creation of the proxy, the proxy might contain a primary key that does not exist in the db. The application should be designed to handle such scenarios when using references. Here is more information on the difference between getReference() and find().
In unit testing, a single component is isolated and tested by replacing its dependencies with stubs/mocks. This allows us to test only the behaviour of the SUT.
Mockito provides multiple methods that help to verify the behaviour of the SUT and also determine how the mocked dependencies are supposed to behave.
verify() this method allows us to verify that a method of a mocked class is called. It can be combined with other methods like times(x) which allowsus to verify that the method is only called x times.
Argument matchers like anyInt(), anyString() and allows us to define a custom matcher using argThat(). These argument matchers can be used to ensure that the correct arguments are being passed to the other dependencies. This is useful if the method you are testing does not return a value useful for determining the correctness of the method.
when() and thenReturn() These are methods that allow us to define the behaviour of other dependencies that are not under test.
For e.g., when(mockLogic.someMethod(args)).thenReturn(value) makes it such that when the SUT invokes someMethod() with args from the mockLogic class, value is will be returned by someMethod(args).
Learnt about how the different features that are provided by GCP and other third parties come together to make Teammates possible.
Most of the information is from the Platform Guide in the teammates-ops repo.
E2E testing complements other form of tests by ensuring that the entire system works as intended and meets the user's requirements and expectations. E2E testing involves testing the application in a production-like environment or as close to it as possible. The complete application is tested from start to finish and it ensures that the application functions correctly from the user's perspective, including all the steps involved in completing a task or workflow.
Page Object Pattern is used in TEAMMATES to facilitate UI changes. In this pattern, a class is created for each page . It helps separate the details of the interactions with the webpage with the rest of the test. Page Object provides an interface for the tests to interact with the page's UI without having to directly manipulate the HTML elements.
Resources:
Mock objects can isolate the component being tested by replacing actual dependencies with mocked ones that simulates the behaviour of the real ones. In this way, the unit test can focus on testing the function of a single component without involving the entire system.
We can use the Mockito framework in Junit tests.
Use the mock() method to create a mocked object of some class. Once created, a mock will remember all interactions. Then we can selectively verify whatever interactions we are interested in.
We can verify the number of invocations of a method using verify(). For example:
verify(accountsDb, times(1)).getAccountByGoogleId(googleId);
We can force a method to return a specific value with stubbing. For example:
when(accountsDb.getAccountByGoogleId(googleId)).thenReturn(account);
Resources:
Snapshot tests are a very useful tool when we want to make sure the UI does not change unexpectedly. Hence, when changing the UI, we need to regenerate the snapshots and commit them.
Generated snapshots do not include platform specific or other non-deterministic data. We can use mock or spy on calls to the class constructor for ES6 class and all of its methods using Jest.
Resourses:
Entity lifecycle
Hibernate can help manage objects and automatically propagate the changes to database. Hibernate entity lifecycle state explains how the entity is related to a persistence context.
persist need to be called toJPA & Hibernate Annotations
Annotations are used to provide the metadata for the Object and Relational Table mapping directly in the Java source code. Annotations such as @Entity, @Column, @Table, and @OneToMany, can define the structure of the database schema.
They are also important for enhancing performance. For example, a join column if not specified with a lazy fetch strategy will cause unnecessary fetch if the data in the join column are not often needed. (Similar problems were presented during data migration some annotations had to be changed in the migration branch).
Resources:
Techniques such as batch processing can minimize overhead and maximize throughput during data transfer. Avoiding unnecessary joins and fetches are also important. For example, EntityManager.getReference can be used to lazily fetch the foreign key referenced object.
topological ordering of the tables being migrated should be established and migrate entities in order.
large-scale data migration needs to be patchable as it needs to be executed multiple times to minimise downtime.
the initial step involved setting up a Google Cloud SQL instance for testing to provide insights into the real-time performance and scalability.
Resources: