Overview
Copado Essentials+ is carefully crafted to address a number of key challenges faced by many Salesforce development teams. After completing the setup of Essentials+, you and your team will enjoy the following benefits:
Complete version history for every changed component for compliance audits and troubleshooting
Prevent team members from overwriting each other
Ability to rollback a deployment
Streamlined change review and approval process
Allow multiple changes to be merged, validated and ready for release days
Support for complex metadata types like Profiles and Permission Sets.
Pre-production deployment reporting.
Enable multiple project teams to work in parallel.
Eliminate hours spent in manual coordination and troubleshooting of deployment errors.
No more month-long releases. Essentials+ enables features to be released daily or as soon as they are accepted in UAT.
A simple and intuitive UX that empowers admins to work with source control seamlessly in a declarative fashion.
A fully streamlined and automated release process that allows your team to focus on delivering business value faster.
This document describes the process flow and a step-by-step guide to setting up Essentials+ to manage your development pipeline.
Choosing A Deployment Method
Copado Essentials offers two different deployment methods to manage your DevOps process:
Deployment Pipelines with Work Items
Deployments with Pull Requests
Note that it is possible for a team to use both methods, depending on the situation.
Deployment Pipelines with Work Items is ideal for teams that share a development environment (e.g., work in a full sandbox) and therefore have a minimal risk of downstream merge conflicts. Work Items simultaneously deploy to the target org and to the target branch in your Git repository. This method is easier because it has fewer steps. Teams with declarative developers that are not familiar with Git, may want to use Deployment Pipelines with Work Items, instead of Deployments.
The second Essentials+ deployment method, Deployments with Pull Requests offer teams that assign developers their own developer sandbox an opportunity to check for merge conflicts using a Git “pull request” to do a code review and approval, before deploying to the target organization. This method is most appropriate for teams or team members that are familiar with using Git repository functionality.
How to decide which method to use. Note that you can mix and match.
Deployment Pipelines With Work Items | Git-based Deployments |
|
|
Configuring Your Git Repository
The process below takes approximately 30 minutes to set up.
What you need to get started:
A Git repository. Essentials+ supports any cloud or behind-the-firewall Git repository including Copado Version Control, Bitbucket, Github, Gitlab, or Azure DevOps.
An Essentials+ (or trial) account.
Step 1 - Connecting orgs to your Git repository
Sign in to Essentials+ with your Salesforce login using a common org across the team, typically your production org (this allows the team to all be represented for simple account management from the Team tab of Account Settings).
Add your orgs to Copado Essentials:
Click Add Organization. For each org that is part of your development pipeline. Give it a name. Select Salesforce Org under type. Under Environment, select if the org is a production level org (signing in from login.salesforce.com) or a sandbox org (signing in from test.salesforce.com). Click Authorize and sign in to the org. Click Save. Perform these steps for up to 50 orgs (per user).
Ensure “Enable Direct Commit to Branch” and “Enable Work Items” are checked:
Navigate to Account Settings by clicking the user icon at the top right corner of the app and then clicking Account Settings. Click on the Team tab in the left hand column. Find and check the two checkboxes on the Team page.
Initialize Your Git Repository:
Depending on the current status of your Salesforce environments, initializing a Git repository can be achieved in two ways. To initialize a Git repository for environments that are closely synced, a single main (sometimes referred to as "master") branch can be seeded, followed by creating new Git branches as copies of the main branch. To initialize a Git repository for environments that are significantly out of sync, each Git branch created will need to be seeded separately to ensure branches most closely resemble their corresponding Salesforce environment.
Git initialization for Synced Environments:
Git initialization for Synced Environments:
Navigate to your Git provider and create a new repo within your account. Once the repository has been created, since a branch is only created once files are committed to the repository, metadata will need to be deployed from Copado Essentials+ to create the main branch. Ensure that your permissions within Git allow for third party tools to perform actions.
Inside Copado Essentials, navigate to the Organizations tab and click Add organization. Select the appropriate Git provider (e.g., Github, Gitlab, Bitbucket, Azure DevOps) from the Type list. Give your repository a name. Click Authorize. This will pop open a screen to log into your Git provider. Once logged in, click the blue “Select Repository” button next to Git repo URL and select the appropriate repository that you just created.
NOTE: If using an enterprise repository (i.e. on-premise), you will need to provide Copado Essentials+ with a Git Repo URL where it can access your repo, and if using HTTPS, a Login Username and Password. If using SSH, you will need to provide your login username as well as configuring SSH keys within your Git provider settings. Copado Essentials+ will generate the private and public keys for your use when you click the “Generate Key Pair” link next to SSH.Click Save on the organization.
Click the Deployments tab along the top.
Create a new deployment by clicking “New Deployment” in the top right corner. Give your deployment a name (i.e. Git Init). Set your Source Org as the top level production org of your development pipeline (this will be your main branch). Set your target org as your Git repository that you added in previous steps. Click Save.
From the Add Components tab in your deployment page, click the component Type dropdown and select Wildcard (the component types are in alphabetical order). On the right-hand side of the component grid, click the Select checkbox that tops the column second from the right. You should see all of the wildcard components selected in this list.
Warning: If you encounter the following error:"Error: Error while retrieving metadata from the source.org. LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 1000" and my commit failed, you will need to break up your deployment of the wildcards into several smaller deployments as opposed to what was recommended above in selecting all the components in the list. Salesforce limits of metadata package zip size is 39MB.Next, go to the Deploy Options tab, scroll down to the bottom of the page and click the “Change” button and click on the gray “Advanced Options” dropdown, as well, to display all of the available deployment options. Find the Git Branch section. Check the box next to “Commit directly to the target branch main”. This commit will now be performed directly to the main branch of your repository. Click Save at the bottom of the page.
Scroll back to the top of the page and select “Commit to Git Branch”. This could take several moments depending on the size of your Salesforce org as Copado Essentials is capturing a snapshot of your org as it currently stands.
Navigate back to your repository. Clone your main or the main branch of your repository for each of your Salesforce orgs that are part of your deployment pipeline. For example, in a pipeline where developers are working in their dev boxes and then merge into an Integration Sandbox or UAT Sandbox, your repository would have a main branch and a UAT or Integration branch. These extra branches allow for you to resolve any merge conflicts long before the changes reach your production environment or main branch.
This is how it will look in Github (the idea is the same in other Git repositories):
From here, select the main branch button, type in your branch name (e.g. UAT), and click Create branch from 'main'.
On completion, the org list should look like below in the Essentials+ app:
Git Initialization for Out of Sync Environments:
Git Initialization for Out of Sync Environments:
Navigate to your Git provider and create a new repo within your account. This repository must be created with the README file included for main branch creation. Creating the main branch with the README file will allow for additional branches to be created from this main branch before having the branch seeded with Salesforce metadata. Ultimately, this will allow for each branch to be seeded individually with the metadata that the branch corresponds with. Ensure that your permissions within Git allow for third party tools to perform actions.
Inside Copado Essentials, navigate to the Organizations tab and click Add organization. Select the appropriate Git provider (e.g., Github, Gitlab, Bitbucket, Azure DevOps) from the Type list. Give your repository a name. Click Authorize. This will pop open a screen to log into your Git provider. Once logged in, click the blue “Select Repository” button next to Git repo URL and select the appropriate repository that you just created.
NOTE: If using an enterprise repository (i.e. on-premise), you will need to provide Copado Essentials+ with a Git Repo URL where it can access your repo, and if using HTTPS, a Login Username and Password. If using SSH, you will need to provide your login username as well as configuring SSH keys within your Git provider settings. Copado Essentials+ will generate the private and public keys for your use when you click the “Generate Key Pair” link next to SSH.
Click Save on the record.
Navigate back to your repository. Since the main branch will be used to serve as a backup for Production metadata, clone the main main branch of your repository for each additional Salesforce org that is part of your deployment pipeline. For example, for a three stage pipeline Dev -> UAT -> Production, a branch will need to be created from main for UAT and Dev, for a total of 3 branches. These extra branches allow for you to resolve any merge conflicts long before the changes reach your production environment or main branch, and serve as a backup when needed.
This is how it will look in Github with only a README file within the main branch (the idea is the same in other Git repositories):
From here, select the main branch button, type in your branch name (e.g. UAT), and click Create branch from 'main'. Notice that the branches being created only contain the README file as these will be individually updated from their corresponding Salesforce environments. This will ensure that the repository accurately reflects how each environment currently exists.
Once a branch has been created for each Salesforce environment, return to Copado Essentials+
Click the Deployments tab along the top.
Create a new deployment by clicking “New Deployment” in the top right corner. Give your deployment a name (i.e. Git Init). Select the source org as your Salesforce environment (i.e. Production, UAT, Integration, etc.) and target org as your Git repository that you added in previous steps, and choose the branch corresponding to the Salesforce environment selected as the source. Click Save.
From the Add Components tab in your deployment page, click the component Type dropdown and select Wildcard (the component types are in alphabetical order). On the right-hand side of the component grid, click the Select checkbox that tops the column second from the right. You should see all of the wildcard components selected in this list.
Warning: If you encounter the following error:"Error: Error while retrieving metadata from the source.org. LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 1000" and my commit failed, you will need to break up your deployment of the wildcards into several smaller deployments as opposed to what was recommended above in selecting all the components in the list. Salesforce limits of metadata package zip size is 39MB.Next, go to the Deploy Options tab, scroll down to the bottom of the page and click the “Change” button and click on the gray “Advanced Options” dropdown, as well, to display all of the available deployment options. Find the Git Branch section. Check the box next to “Commit directly to the target branch main”. This commit will now be performed directly to the main branch of your repository. Click Save at the bottom of the page.
Scroll back to the top of the page and select “Commit to Git Branch”. This could take several moments depending on the size of your Salesforce org as Copado Essentials is capturing a snapshot of your org as it currently stands.
Deployments can run concurrently, so repeat the Deployment setup and commit, changing the source org and target Git branch to reflect all other environments that will be utilizing the repository.
On completion, each Git branch should accurately reflect the metadata as it exists in the corresponding Salesforce environment.
Step 2 - Sharing orgs and repositories
Sharing orgs and repositories reduces repetitive work. This Help article called Team Collaboration provides instructions for setting this up.
Configuring Deployment Pipelines with Work Items
Salesforce developers and admins count on Copado Essentials to quickly and reliably deploy changes from one Salesforce org to another. As teams grow in size and complexity, it is imperative for teams to go beyond org-to-org deployments and start thinking about how to formulate a consistent release process with good control and governance in place. This is what Deployment Pipelines with Work Items was built for.
The Deployment Pipelines with Work Items method
The Deployment Pipelines with Work Items method works like an org-to-org deployment except that deployed components are simultaneously committed to your Git repository to provide backup and version control. A Work Item is an encapsulation of a user story and its progression through a consistent change management process with approval, history, visibility and collaboration built-in.
Turning On Work Items
Deployment Pipelines with Work Items are not enabled by default and will need to be turned on by the Team Owner of your account. If you're the only person on your Copado Essentials team, you are the owner. If you aren't assigned the Owner role on your team, please ask the Owner with permissions to activate it for you. To activate this feature go to your Account Settings. Under Settings visit the Team section and check "Enable Work Items".
Essentials+ Trial License
The Deployment Pipelines with Work Items feature requires Essentials+ licenses. However, you can enable Work Item at any time without an Essentials+ license for a trial period of 21 days. After the trial period ends, you need to upgrade to an Essentials+ license in order to continue using Work Items.
For details about how the Essentials+ Trial experience compares to a paid Essentials+ license, check out this document below.
Setting up your deployment pipeline
To get started, sign in to Copado Essentials+ as a user with the team owner role. Navigate to Your Profile > Account Settings > Application Settings > Pipelines > Add Pipeline.
The Add Pipeline link is in the upper right corner.
For each environment in your pipeline, enter the environment name and associate a Salesforce org with that name. Note that you may want more than one pipeline. You might have one for standard (forward) deployments, one for back deployments, etc.
After the pipeline is saved, it will be visible to all members within the team. However, before other members can use the pipeline, the owners of the associated Salesforce orgs in the pipeline must share org access with other team members.
Enable version history tracking (optional)
With Deployment Pipelines with Work Items, you can enable version history tracking for your pipelines. For pipelines that have version history tracking enabled, Copado Essentials automatically commits any deployed metadata changes to the configured Git branch as soon as your work item is deployed successfully to the corresponding environment.
To enable version history for a pipeline, simply edit the pipeline, select a Git org and a corresponding branch for each environment, as shown below.
Create Work Items
Any member of the team can create work items. To create a work item, navigate to Work Items >
New Work Item.
Work Items list view.
New Work Items edit screen.
Users are only required to enter the “Issue” field.
“Pipeline” field is automatically populated with the default pipeline configured by the team owner.
“Start from” environment is automatically populated based on the pipeline setting but users can choose to change the “Start from” environment.
Deploying Work Items Through a Pipeline
Users can perform various actions with work items including compare changes, add components, share deployment, or comment on a deployment.
Work Items metadata components list with pipeline stage progress at the top.
On successful deployment of a stage, the work item automatically progresses to the next stage.
If version history tracking is enabled for the pipeline, Copado Essentials automatically commits any deployed metadata changes to the corresponding Git branch. You can view the details of the commit/history by clicking on the Git branching icon next to each environment.
Optionally, users can go back to a previous stage if needed by clicking on More > Go Back to Previous Stage.
Permission levels (validate, deploy, view, manage) are applied at each stage of the pipeline based on the org sharing configuration.
Upon successful deployment of the final stage, the work item is considered complete and closed.
Configuring Deployments with Pull Requests Process
Key steps
Admins/devs build and test features in their dev sandboxes.
The admin/dev then submits a pull request against the UAT branch via Essentials+ Git flow. Once the pull request is approved and merged, Essentials+ CI job picks up the changes and kicks off the incremental CI build to deploy the feature to the UAT environment.
Once the feature is tested and accepted in UAT, the admin/dev submits a second pull request targeting the main branch.
Notifications are sent to the team leads for approval. Upon approval, the changes are queued up in main ready for production deployments. Production deployments can be triggered automatically via webhook, time schedule or manually.
Deployment with Pull Request method deployment flow:
The Deployment with Pull Request method deploys through your Git repository to allow creating a pull request before updating the target branch and org. This provides an opportunity to check for merge conflicts.
Notes
In the diagram above, there are two main branches: main and UAT. Any commits to the main branch get deployed to your production org. Any commits to the UAT branch get deployed to UAT org.
The CI jobs are set up as incremental builds which deploy only changes since the last successful build rather than deploying the entire branch. This helps to speed up deployments as well as minimize the risk of overwriting unrelated changes in production.
Wildcard allows us to select all components of a certain type without having to name each component explicitly. You can add wildcards via Add Components > Type > Wildcard. We recommend starting with Code Related, Object Related and Process Automation groups. You can add more metadata types at a later stage. Do not add Profiles at this stage, see Working with Profiles section to learn more.
Under Deploy Options, set Source Format to Salesforce DX. See Source Format Selection to understand the reasons.
Hit Commit To Git Branch and merge the changes to main.
Now that your main branch reflects production. Create a UAT branch based on the main branch and follow the same step above. This time, change the source to UAT org and target branch to UAT branch.
Step 3 - Setting up CI jobs
Create a new CI job that deploys from UAT branch to UAT org, like below.
IMPORTANT: After saving, edit Deploy Options and change the Git Source option to deploy files committed since the last successful build, like below.
By default, the last successful build marker points to the head of the branch. If you click on Review Pending Changes, the queue should contain zero components at this point.
Now that your UAT job is all set, you can create the Production Validate job by cloning the UAT job and change branch to main and the target org to Production org, like below:
The above CI job validates changes against Production as soon as a commit is merged to main. If you want to start with a manual trigger instead, simply uncheck the webhook trigger checkbox. You can always enable this later.
Step 4 - Submitting your first change request
Create a new deployment from your dev sandbox to Git targeting the UAT branch, like below:
After saving, add the changed components and click ‘Commit to Git Branch’.
Tip: you can validate your selected components against the target Salesforce environment before committing the changes . This can be done via More > Validate.
After the commit operation completes, click Review Changes and submit the pull request. After the pull request is merged, the change should be deployed to UAT automatically by the CI job.
Once the change is tested in UAT, clone the deployment and follow the same step above. This time, change the target branch to main in order to have the feature released to production.
After the pull request is merged to main, review your Production CI job by clicking on Review Pending Changes. The diff report should show the files that are about to be deployed and the differences against production.
Hit Deploy or Quick Deploy and the feature should be deployed to production.
Fine-tuning
Understand incremental diff deployments
An incremental build contains only files committed since the last successful build which is indicated by a build marker. Every time a deployment is successful, the build marker is automatically moved to the head of the branch.
This means the pending change queue is reset after every successful deployment.
No manual intervention is required in general. However, if you ever need to change the build marker to a specific point, simply edit Deploy Options and point to the new commit id.
Note that for an incremental validate CI job, a successful validation does not move the build marker automatically. The build marker for a validate CI job is only moved when the equivalent deploy action is executed successfully.
Working with profiles
A full Profile file can contain 50k lines of xml which makes it unfit for the purpose of source control driven deployments and merge resolution. In order to work with profiles effectively, commit only the base profiles when you initialize your Git repository for the first time.
You can commit only the base profiles by creating a separate commit deployment from your Prod org to your Git repo and include only the Profile wildcard (excluding any other wildcards), like below. This way, the profiles will not carry any other related permissions.
After a few months of development, your profile files will eventually grow to a certain size. At that point, you can decide to trim the profile and bring it back to its starting baseline by repeating the above commit deployment, this time, edit Deploy Options and change the Git Commit option to Overwrite.
Overwrite means that Essentials+ will simply take the base profile files and overwrite the large profile files in your Git repo.
Source format selection
We recommend to use Salesforce DX source format instead of the old Mdapi source format for the following reasons:
Incremental diff deployment works better with DX source format. When you commit a single field, the diff would include only this single field in the incremental build. In the case of Mdapi format, since the field is part of a big object file, the entire object is included which contains a lot more than just the single changed field.
The breakdown DX code format makes merging easier.
DX source format is a prerequisite for developing Lightning Web Components.
Note that DX is a spectrum of technologies including a new code format, CLI, scratch org, source tracking, and packaging 2.0. At this stage, the only part of DX that we recommend teams to leverage is the new code format. The remaining parts still need a lot more time to reach production maturity. Scratch org and packaging 2.0 requires a significant amount of upfront investment in breaking down the org into modules and the overhead of managing dependencies between these modules cannot be understated.
Switching from Mdapi format to DX format does not change your development workflow. You can continue to develop against sandboxes (instead of scratch orgs) or use your favourite developer tools (VS Code, Illuminated Cloud, Welkin Suite). All these tools now support using DX format against sandboxes.
This does not mean that you cannot use the Mdapi format. Essentials+ supports both DX and Mdapi source format. Just keep in mind that any deployment strategy that involves calculating differences would normally work more effectively when the source is broken down into smaller files.
Understanding the inclusion option
Using an incremental deployment strategy is a great way to speed up your CI builds. Occasionally, there could be specific scenarios where you might want to include certain files in your CI builds even if they have not been changed. This can be achieved by leveraging Essentials+ Inclusion option.
Simply edit the Deploy Options, and enter paths for the files you wish to include in the CI build and these files will be included together with the result of the diff calculation.
[End]