Comments on Lessons Learned

During the development process, the SRC and NSF teams have conducted weekly collaboration meetings to review status, demonstrate features, discuss changes, and obtain detailed information to support the development effort.  The Lessons Learned listed in this document are based on SRC’s development and comments made during the weekly collaboration meetings.  Additional lessons learned may be provided by the Government stakeholders based on their perspective.

1.1 Collaboration Meetings

Weekly collaboration meetings have been a key to fully understand requirements, stakeholder priorities, and potential issues as early as possible.  Regular meetings should be continued during the development process.

1.2 Feature Demonstrations

Screen share using MS Teams has been used for meetings and to demonstrate features and items under development.  The use of a screen share combined with discussions allows current status to be demonstrated efficiently and allows stakeholders to be involved in development decisions as early as possible.

1.3 User Feedback

After initial development was performed and some fields for user entry were created, Government users were allowed to access the system and provide feedback.  This feedback was used to enhance the UI/UX and suggested changes were implemented.  Using Government users to test the system should be continued to obtain feedback from outside the development and stakeholder team.

1.4 Development Process

The Salesforce tool used as a basis for the Standard Application Process lends itself to rapid configuration and prototyping.  This allows weekly updates to be demonstrated that are functional without relying on a traditional software release development cycle.  Rapid prototyping and continuous demonstrations should be continued.

1.5 Financial Reporting

Financial reports are provided on the 1st of each month.  Due to the delay in processing timekeeping systems, actual information is not available until the 7th of each month.  This causes financial reports for the 2nd half of each month to be estimates based on individual employees providing hours charged to the project.  Because final information is not available until the 7th.  The due date for financial reports, and possibly status reports should be scheduled for the 10th of each month following end of the reporting period.

1.6 User Account Creation for Testing

A second round of user testing was conducted for external users to act as applicants.  During the testing, some users had issues receiving email notifications and creating a user account.  In some cases, testers already had internal user accounts.  Due to the issues, many testers were unable to provide feedback in the limited test time.  An alternative process was discussed and will be implemented for testing the review process.  The alternative process is to create user accounts with passwords and assign them to testers.  This will allow testers to focus on the process rather than wait for accounts to be created.

1.7 Additional Features and Requests

During the development period, stakeholders have become more familiar with the capabilities of the tool and developers have learned more details about the overall application process.  This has led to discussions about including additional capabilities and the effort required to deliver ‘nice-to-have’ items. Although not specified in the initial overall requirements or identified as non-specified derived requirements many of these features can be included with a small resource investment.  Because the development effort has been held under budget, resource funding is available to include these items.  It is valuable to have a risk management or contingency line item to plan for changes or updates during the development effort.

1.8 Configuration vs. Development

SRC chose the Salesforce database as the underlying tool for the Standard Application Process portal.  A significant feature of Salesforce is the no-code/low-code capability to implement features.  It is not uncommon for projects requiring software development to be over budget and delayed on schedule.  A frequent cause is due to an insufficient understanding of requirements and underestimating the effort required to create custom software to meet requirements as they are being fully understood.  The no-code/low-code capability allowed the SRC team to focus on configuration of the tool to meet specific requirements rather than requiring custom software development.  This has led to the ability to rapidly adjust processes based on user feedback and with the expectation that a detailed understanding of requirements would lead to specific implementations.  Overall, this has led to a flexible tool that has been adjusted for specific user requirements without requiring additional funding.

1.9 User Licenses and Access

The Salesforce application is delivered with a renewable license model on a 12-month duration.  The current licenses expire on 3 November 2024.  Although this is 5 weeks after the end of the Period of Performance, the time may not be sufficient for Government evaluation and transition if the solution is chosen for the production effort.  Future proposals should include licenses to support the development effort with a single ‘continuation’ license to maintain the system for 1 year after the conclusion of the Period of Performance.  This would allow sufficient time for Government evaluators and a transition to production.

1.10 Capabilities Deliverable

Because capabilities were added throughout the development effort, a final listing of the system capabilities was not available until the completion of the project.  The final user guides provide descriptions of the capabilities and will be included as deliverables.  The benefit of agile development to modify the desired capabilities was determined to be more important than an early description of capabilities prior to prioritizing the separate features.  For future projects, a short bullet list of high-level features should be included at the start of development with full capabilities and features described at the end of the effort.

1.11 Hand-off to Government

The current hand-off of the prototype to Government users is informal on a specific date. Future efforts should include a checklist of items and a specific transition day.