If you've recently added Copado Essentials Data Deploy to your DevOps toolkit, you might be wondering what your next steps are. Well, in this article we'll help to outline what it takes to get started deploying data up to Production, or down to your Dev environments. What we'll cover below will take us through downloading and importing templates, configuring those templates for your specific use, what you'd need to do differently to seed a sandbox, and how to deploy your data.
Downloading and Importing Templates:
If you've purchased Data Deploy for the use of moving reference data, the first step you'll need to take is to visit the Copado Success website to download the Data Templates for your managed package (long gone are the days of using Data Loader and spreadsheets to maintain your object relationships!). After creating a Copado Community account, navigate to the DevOps Exchange to select the suite of templates that apply to you (whether that be for CPQ, nCino, Conga, or Veeva), then simply download these JSON files for import into your Copado Essentials account.
To import these templates, login to your Copado Essentials account, select "Data Templates" from the dropdown under the Data tab, and click the "Import Data Templates" button on the top right. From here, you can choose to import any selection of your templates, and after import, you'll see a template for the main object and sub-templates for all child and parent objects referenced by the main object template.
Configuring Pre-Built Data Templates:
Once your templates have been imported you'll be able to customize them to fit your specific use case. By default, Object Fields, Parent Objects, Child Objects, and Parameters are all pre-selected to accommodate out of the box use, but you're free to add or subtract any fields or objects as they relate to your needs. As a precaution, within the Object Fields tab we highly recommend utilizing the External ID field (or the Lookup Key for nCino users) to ensure that an upsert takes place, so duplicates aren't created in your target environment during the data deployment process. This can be utilized by adding an External ID field to all objects referenced within your template and sub-templates, followed by selecting the field as shown below.
If you don't have any values within your External ID field, we can easily create a value for you by using the Auto Generate feature located in the "Field Content Update" dropdown on the right, then allowing you to select up to 3 object fields to hash for a unique value.
When it comes to selecting Parent and Child objects to be included in the templates, our pre-built templates come with all of the standard relationships for CPQ, Conga, nCino, and Veeva already preconfigured. However, you have the option to adjust these for any purpose that you need. To do so, just navigate to the Parent Object or Child Object tab to choose the objects you'd like to select or deselect, along with assigning a sub-template for any object added.
As a way to further customize these templates, you do have the option to apply a filter to the main object using the Main Object Filter tab. Here, you can add parameters and filter logic to specify which records you want deployed. When defining parameters, we recommend restricting this to only the top level object in your template, as adding parameters in sub-templates will significantly decrease the number records that fit your specified criteria. Parameters can be set for any field within the main object, and for more unique filtering situations, you can use a custom filter to reference cross-object.
Once your main object template has been configured to your use case, you can review the relationships visually by using the Relationship Diagram tab. This viewer will present the same information that you specified on previous tabs in a much more visually appealing way, helping to portray the relationships as they are currently defined, as well as indicating which objects have filters and files (see Sandbox Seeding Basics) attached to them. Once the template has been tailored to your needs, click "Save" and "Activate" for immediate use (all sub-templates/related templates must also be activated).
Sandbox Seeding Basics:
If you've purchased Data Deploy for the purpose of seeding test data into Developer environments, then you'll likely need to create your own Data Templates. To get started creating a template, login to your Copado Essentials account, select "Data Templates" from the dropdown under the Data tab, and click the "New Data Template" button on the top right.
The first tab you'll be directed to is the Template Details tab. Here, you can define the source org of your template (which is only used to analyze the schema of your objects--this is not necessarily the org your data will come from) as well as the Main Object. You're free to select any standard or custom object as the main object for your template. In addition to specifying the source org and main object, you can define if you want your deployment to proceed in the event of an error, and how you want any attachments to be handled during a deployment (inserted, replaced, removed, or none of the above).
As detailed above, the Object Fields, Parent Objects, and Child Objects tabs can be used to create a web of relationships for records you want to be deployed from your source org to your target org. Within the Object Fields tab, we highly recommend utilizing the External ID field to ensure that an upsert takes place, so duplicates aren't created in your target environment during the data deployment process. This can be utilized by adding an External ID field to all objects referenced within your template and sub-templates. To select Parent Objects and Child Objects, navigate to the corresponding tab to view the auto-populated list based on the Main Object selected. For every object that is selected on either of these tabs, you will need to create a new sub-template (or select one that was created previously) to specify which fields of a Parent or Child object you'd like to deploy.
Once your relationships have been defined, navigate to the Main Object Filters tab to specify parameters, filter logic, and record limits. You can deploy up to 40,000 records using a data template with one main and no related objects, and up to 8,000 records using a data template with one main and many related objects. As a final step, you can review your template using the Relationship Diagram to ensure the relationships shown match what you've intended. Once you're satisfied with your template, click save and activate your template for immediate use.
Deploying Data:
After defining your templates, the last step is to deploy your data. To ensure a successful deployment, there are two pre-deployment actions that are recommended. First, the metadata within your source and target environments need to match, so a metadata comparison between environments is recommended. This comparison can be accomplished by setting up a mock metadata Deployment and choosing the same source and target orgs that will be used in the Data Deployment, then comparing the fields and objects that data will be deployed between. Second, it is best practice to disable Salesforce Triggers, Workflows and Validations when working with Copado Data Deployments. These processes may adversely affect the insertion or update of a record, so temporarily disabling them is recommended. For more information on how to disable these processes, see this Copado Enterprise help article.
When you're ready to deploy your data, click "Data Deployments" from the dropdown under the Data tab in the top banner. On this screen, you'll select "New Data Deployment", after which you can define your source org, target org, and data template to be used. When multiple team members are using Data Deploy you can easily share templates across accounts by activating the "Include team templates" checkbox, as shown below.
Once the deployment has been defined, click save, followed by the Deploy button, to initiate your data deployment. After your deployment has completed, you'll be able to view details relating to the deployment by clicking the blue button beneath the deployment status.
On the view details pop-up, you can see how many records were deployed, and from which objects, to double check your object relationships were maintained as intended. If there were any failed record deployments, you could see those indicated in the center column, and get additional information by downloading a detailed CSV of the deployment using the "click here" button.
Data Sets:
Currently, the importing and exporting of Data Sets is not supported with Copado Essentials Data Deploy, however this is on the roadmap for future releases. This feature is currently available with Copado Enterprise, for which you can review the documentation here.
Conclusion:
Data Deploy is a powerful add-on that can help to simplify and automate your data migrations, all while complementing the quick metadata deployments of Copado Essentials. If you need any additional resources while setting up and using Data Deploy, feel free to reach out to us in-app using the chat icon on the bottom right for access to 24/7 support.