Doctrine, Symfony & the Database

Hey EinzigTech!

Ah yes - this is a complex topic :). Let me answer your question directly first:

Adding a property/field to an object is not the issue, but will those fields be persisted into the database (assuming that the related column for these dynamic fields somehow exists in the database)?

Yes and no :). If you do manage to add a field to the PHP class and add the column to the table, Doctrine will not see it automatically. However, if you build the cache, then it will see it. Basically, Doctrine caches its "mapping metadata" (e.g. the annotations above each field) as well as its query building (I think you will need to rebuild both... but maybe only the mapping metadata). If you're using the latest DoctrineBundle, then the cache is in a "cache pool" called "cache.doctrine.orm.default.metadata" - you can find a full list of cache pools by running "php bin/console cache:pool:list". You can clear an individual pool by running php bin/console cache:pool:clear the_pool_name.

The less direct answer to this question is that... people don't usually do this. Typically, if you need super-dynamc properties (like, an admin is able to create properties via an admin interface), then you use an EAV pattern - https://stackoverflow.com/questions/14123953/doctrine-orm-on-eav-tables - but your use-case may be unique, so I'm not sure :).

Cheers!

Hi weaverryan ...

Thanks for your response. I am really in a fix with this Doctrine issue. I am using MongoDB ODM (MongoDB Bundle) and as NoSQL don't need an exact schema, I was trying to dynamically set the properties to the Document. I am also using Api-Platform for the project. One of my approach was to use DTOs and set dynamic properties there. Then collect them all and set it as a HASHed property name `data` in the document. I am not too happy with this approach as our Data Scientist will have direct access to the database for getting the data, so I would prefer to have all fields separately rather than putting them as a json object (hash) in the `data` field. Also, for outputting the resource, when I use an output DTO and add the fields dynamically to it, all the added fields are just ignored. Now the situation is becoming such that, the team is trying to shift away from Symfony to Laravel (which I personally would not like as it then the followup projects will also be in Laravel).

Best regards...

Hey EinzigTech!

Hmm. So it's still not clear to me: who/what determines these new fields? What I mean is, if you're adding properties dynamically at runtime, what is causing that? Is there an admin that clicks through an interface and selects "add new field"?

Anyways, I think part of the problem you're running into is that API Platform is trying to very "static" on purpose. One of its jobs is to generate API documentation. But in your case... if you got everything setup correctly, the API documentation (and API) would dynamically change from day to day. Today there might be a "foo" field, which I program my API client to look for, and tomorrow, it's gone - but there are 3 new fields. That's just... super weird :). Also, this is one, sort of, "knock" against using a schemaless database with an ODM in general: if you truly need a schemaless database, then trying to map it to a concrete class is a weird situation.

So, I recommend backing up a bit. If you have a truly dynamic schema - i.e. one that will change even without you deploying new code - then you should not use an ODM. Or, perhaps, maybe you have the ODM with some "base" properties to make your life easy in some cases... but ultimately, you should query Mongo directly for data. As far as API Platform goes, I'm not sure it's a good fit for this. Or, perhaps, it may be a good fit IF you store all your extra, dynamic fields in some dynamic "data" property. From an API perspective, this makes sense: we need our API to be dependable, to have all the same fields today as it does tomorrow. If you have some part that is truly super dynamic, then isolating that into a "data" field that is "a hash of data with variable keys" seems sensible. You could populate that key via a custom data provider I think.

Anyways, let me know if this helps. My guess is that the push to switch to Laravel is because there are less rules over there. And in this case, yea, you kind of DO need less rules - you need to pull back from the structure that the ODM (and API Platform) is putting on you.

Cheers!

Hi weaverryan,

First of all, thanks a lot for taking the time and reading my comments/question/confusions and thinking about them. Our actual plan was, to have a dynamic json input with different fields for different related child objects (apps). One of them (like a parameter named "type") will have a key which determines about the child data. Then we validate the data mapping with some yaml file and convert the data from the DTO to our required database structure. As MongoDb is a NoSQL, this dynamic structure on the database level wouldn't be a problem. One benefit would be, if we needed a different structure for some new child data, we could easily drop a small yaml file and all would have been automatically taken care.

But as it was getting more and more complex, we decided to make it in the normal pattern (everything defined). Now we have collected all the fields and defined them in the input DTO with a default value of null. Then in the transformer, we validate and restructure it based on our requirements and then store it in the database. For output, we show different output DTOs based on the requesting app from the output transformer.

To make it more clear:
Input comes from:
App_1->customer_info
App_2->customer_info

But if this customer is the same, we store it under the same customer document in the database like Customer->app_1_data and Customer->app_2_data

Now if App_1 requests the API for the customer info, it gets the basic customer info with app_1_data and App_2 gets the basic customer info with App_2_data. If the app specific data is not found, it looks for the data in other app_data section to get some basic data.

I am very sorry to post this here, as we started with discussing Doctrine and now it has gone completely in other direction. But this (taking different direction to solve a problem), I think, is a very common issue in agile systems :-P .

Hey EinzigTech!

Thanks for the update - and not worries about posting this here - if chatting to me helped solidify anything in your head, then it's worth it :). The new direction seems - to me - like it'll be much less of a headache - even if it's still a really complex problem ;).

Good luck!

Hi Andres M!

Yes, you're right! So, there are 3 different types of caches in Doctrine: metadata, query and result caches. The first 2 - metadata & cache - make complete sense to have enabled all the time in production - they cache the Doctrine annotations and how it *builds* the queries (not the results). But thanks to the prod config that you get out-of-the-box - https://github.com/symfony/... - this is already configured for you. So, you don't need to think about it.

The 3rd type of cache - result cache - is something that you need to opt into when you make a query. Basically, you tell Doctrine: "please cache the result of this query". I've actually *never* used it before. There's not a problem with it, but since Symfony already has a great cache system, if I need to cache anything (whether it's data from Doctrine or some other data) I put that data into Symfony's cache system. It's one cache system for anything I need :). We use it in our Symfony fundamentals course: https://symfonycasts.com/sc...

So that's why you don't see anything here :). But if you have any questions or doubts, let me know!

Cheers!

Hey!

We're so glad to hear that you find our tutorials useful. At this point we do not have planned to create a tutorial about DDD or microservices but those are topics that we have in mind. This kind of feedback help pushing forward the release of those tutorials :)

Cheers!

Thanks for getting back to me!

Awesome, I'm happy that they are being considered :) hopefully they will come soon

Hey @kibria4!

As Diego mentioned, those topics *are* already on our list, but not officially being worked on. But since you are already studying them, I would *love* to know what specifically you're interested in about each of them - especially microservices, which can be a *huge* topic (communicating between microservices, deployment, authentication, organizing a micro Symfony app, just to mention a few things that come to mind quickly). That's one of the reasons I haven't talked microservices here on SymfonyCasts yet (even though we have 3 microservices that operate behind our main site): I'm not *exactly* sure what parts are most important to people.

Cheers!

Hi @weweaverryan !

Well I am a beginner to Microservices and DDD pattern so I'm studying about making each service a business domain and then connecting them together with a gateway. Some of the things I'd like to know about are merging data together from each service that have relations and how authentication and authorization works in a microservices architecture. For my upcoming project, I am required to to build a SaaS system that manages shift rotations and keeping a group of users' documents up to date. Due to the size of the business and the features they require, a Microservices architecture is most probably necessary, especially as different groups of users will be using different sections of the system.

I think these topics may be good to help beginners get started on Microservices with Symfony:

- Create a few individual services in Symfony (or API Platform)
- Relationships between services
- API Gateway built in Symfony to connect them, or maybe something that's already out there that is easy to work with, like Tyk (I tried Kong and that was quite a headache to get it running!)
- Authentication and authorization
- Possibly service health check to make sure developers are notified when a service is down or can cause problems to other services
- A frontend to communicate with the backend

I'm having trouble thinking of what should be covered afterwards as not all parts are related to Symfony per se. Now that I think about it, deployment is a big and tricky section as there are multiple hosting providers and each will have their own configuration methods. If I can think of anything else that could help shape a Microservices course on SymfonyCasts, I'll let you know!

Hey Mike,

Yes, unfortunately, no course about microservices yet, but that's a really good topic and we would really like to release a course about it in the future. This topic is in our ideas pool, but I don't have estimations yet when it might be released, sorry. There're a lot of good stuff that we would like to release before start working on this course.

I'm sorry to hear that it's slow for you, but I assure you that we're doing our best on releasing the content. Usually, our solid release schedule is release one video a day. Sometimes we can release a few videos at once, it depends on our progress. Releasing our videos is a complex process, and sometimes it's not clear why 5-minute video takes so long to be released behind the scene. We're doing a lot of planning and draft coding, then we create scripts and record video and audio, then we postpone the video adding some cool animation effects and exclude long pauses to make the video as short as possible, combine audio and video, etc. And finally, we're adding dynamic code blocks to the scripts so you can see the exact code we are using in the course. I completely understand that you want to get videos faster, but that's our best pace we can move for now.

> This tutorials are released so slow that when one is finished a new version is already out

Haha, sometimes that happens :) Well, if we're talking about minor version - most probably it's a coincident ;) If you're talking about patch versions - yeah, that happens of course, depends on the package activity on GitHub. If major version changed - that's a totally bad luck, but usually we avoid such thing, I don't remember when it even happens :) Yes, our courses are releasing in full not in one day, but usually within a month we're completing our tutorials in progress as our statistics show. Our releasing policy is to deliver content as quickly as it's ready. We don't want users to wait 1 month before releasing the course in full the same day. Instead, we release a video or few per day so that our users may watch it faster.

I hope this sheds the light on our release process and is more reasonable for you now. Thank you for your patience and understanding!

Cheers!

Hey Guido!

Not in this tutorial - DTO's aren't really related to Doctrine - they are more important depending on *how* you want to use them - like DTO's in the form component or DTO's with API Platform. What is your interest with DTO's?

Cheers!

Hello Ryan,

first: Thank you for you amazing work!

Many sources say that you should not directly link entities and forms ( https://stackoverflow.com/a... ). This includes ocramius as a doctrine developer. But the Symfony documentation still propagates this. So I would be interested to see more complex examples. Often you will find only one form with 2-3 fields in these examples, but no more complex many-to-many entities are involved.
I would like to see some "best practices" here.

Hey Guido

Currently we don't have a complex case where you show how to manage it with DTO's but you can watch this video to get a better idea of how to do it https://symfonycasts.com/sc...

Also, you may find interesting this talk from SymfonyCon 2018 about rich forms https://symfonycasts.com/sc...

I hope it helps!

Will have a look, thank you!

Sorry, the link contains a ")" at the end (please delete this).

Hey Guido,

Done! FYI, you can edit *your* Disqus comments yourself as well in 7 days after posted them, see "Edit" link below your comments ;)

Cheers!

Was wondering the same thing lol

Hey ian ,

Haha, check this tweet, guys! https://twitter.com/weaverr...

Cheers!

Hey Vladimir,

Haha, stay tuned to know something new! ;) We will start releasing this very soon.

Cheers!

Looks like the generated HTML for multiple checkbox is not goot. It's missing [] behind the name.

I got
name="status[processSteps]"
instead of
name="status[processSteps][]"

This is not due to the simple array type as I just noticed the same problem for an entity type field.

Could it be a bug? I am using vers 5.0.11.

The bug was me. For an obscure reason, I had overiden the twig block "choice_widget_expanded" as follows:

`{% block choice_widget_expanded -%}

<div {{ block('widget_container_attributes') }}>
    {%- for child in form %}
        {{- form_widget(child, {
            <b>full_name: full_name,</b>
            parent_label_class: label_attr.class|default(''),
            translation_domain: choice_translation_domain,
            valid: valid,
        }) -}}
    {% endfor -%}
</div>

{%- endblock choice_widget_expanded %}`

This cause the issue I had ...

Hey be_tnt!

Phew! I'm glad you figured that out - that was tough one! Thanks for sharing your answer in the end :).

Cheers!

Hey It O.!

I just uploaded one this morning - it should be out tomorrow. Then we'll hit the gas on this tutorial :). I was out last week for vacation, which is why the first 2 chapters came out this week, but none yet. Oh, and *technically* the release date is tomorrow even for the first chapters - (that's when we'll announce them on Twitter). We've tweaking our process to get videos released to people on the site more quickly (e.g. last week) even if the official release date for a tutorial or chapter isn't until later :).

Cheers!

Hey It O.

This course is at the oven right now! Only one more week and it will be ready to be consumed ;)

thank's

Hey Mdev,

We're actively working on it right now! I think it should be started releasing in about 2 weeks.

Thank you for your patience!

Cheers!

I just finished the video recording! So Victor is right - 2 weeks or, hopefully, sooner :)

Any chance this weekend?

Maybe yes, maybe no, stay tuned!

Cheers!