Dynamic Forms for Laravel

In my line of work, we have a perennial problem: somebody wants a forms to collect data. Most of Web Development is making <form>s for one person, validating the data, and showing it in a table to somebody else. Some times, people don't want their Web Developer to make a form; instead, they want to make their own forms, like building a SurveyMonkey thing.

For one project, custom forms and workflows¹ were critical. We were building software for both sides of a research grant: departments offer our students research grant opportunities, the students apply, a committee reviews/rates proposals, yadda yadda. We have two dozen grant offers with ever-shifting requirements. Opportunities are sun-setting or coming onboard all the time.

I wanted to give the folks setting up new grants a really powerful tool to set up their application/review/etc forms. But this tool needed to be easy to use. The users aren't Web Developers, they're faculty. They are familiar with SurveyMoney and Qualtrics, so something akin to that would be good.

Enter: Formio

Form.io is a SaaS product for building custom forms and receiving the data in a backend, where you can do Stuff with it. The SaaS part isn't of interest to me, but they're an open-source company:

• formio.js is their JS library for building forms & displaying/validating the built forms
• formio is the backend, for receiving data and actually validating it

Their client-side library offered a solid user experience: drag-n-drop, plenty of settings, and an extensible library of widgets² for the forms. The builder produced a JSON definition of the form, so that's easy to store in the backend and work with.

I spent a while fiddling with the JS library, and eventually figured out how to disable/hide options in the builder. It had a bunch of things like running arbitrary Javascript code for validations, controls for DB indexes, and some HTML-y settings. All of this stuff would just be a distraction to my users.

The other mode, where you give the form a JSON definition of a form, worked great. There was a little bit of custom JS to make it submit to my server and handle auto-saving³ was all it needed.

Validating Data

Widgets can have different validations and calculated values set up, using either point-n-click "Required" checkboxes or basic scripting in JSONLogic⁴.

The client-side JS library can run all these validations and give people instant feedback, but it still needs a way to validate the data sent to the server. Otherwise, people can just POST garbage to the backend, and that wouldn't be good.

I wanted to use the data in my Laravel app, and present it joined together with relevant relational data. I could have run their backend and tied into MongoDB to get the form data out, but it looked like that would make sorting/filtering rows really hard -- much harder than ORDER BY data->'proposal_title' DESC.

In order to validate the data, I needed a library that could read the JSON form definition from the builder, parse out all the rules/calculations/conditional field logic, and re-implement all of that stuff in PHP. For about 20 different input widgets, and another dozen layout widgets.

So I did that.

Dynamic Forms for Laravel

When I say "so I did that", what I actually mean is "so I commanded a team of developers to do that". The initial version was myself and another coworker.

He did a lot of work adapting Lodash functions available to JSONLogic into PHP, so that feature worked as expected. You can define complicated calculations or validations with JSONLogic since it can access all the fields in the form⁵. It gets really powerful with Lodash for map/less-verbose averages/etc.

This took us the better part of a month to build out. Writing tons of unit tests was key here -- that's the only feasible way to ensure we're testing all the different combinations of settings you can have. The validation rules and calculations for one widget are complicated enough, but you can have calculations that touch conditional fields with their own logic and validations and that gets really complicated.

It's been an ongoing project as we need more features or find bugs. Big thanks to the whole team for making this thing work:

We've been using this in production for a couple years now, and it's slowly appearing in other apps -- and in other departments' work! Which is cool.

Using the Data

When a form gets submitted, we store it in a PostgreSQL JSON column. We can generate data tables off our "static" schema (who applied for the grant, yadda yadda) -- and mix in the form fields set up by the grant admin:

SELECT
    proposal.id,
		proposal.submitted_at,
		proposal_author.id AS author_user_id,
		proposal_author.full_name,
		form.data->title AS form_title,
		form.data->abstract AS form_abstract,
		/* etc */
FROM proposals
INNER JOIN users AS proposal_author ON proposals.author_user_id = proposal_author.id
INNER JOIN form ON (proposals.id = form.proposal_id AND form.proposal_form_type_id = ?)
ORDER BY form.data->title ASC

This is a simplification, but this is the big reason I wanted this data to live in my app: querying, sorting, filtering is all trivial to do this way. There's a ton of code to build the query out with all the right fields based on the form builder's JSON⁶, which produces some monsterously complicated SQL -- but the logic for building it isn't hard.