Handling SQL updates for production deployments Announcing the arrival of Valued Associate...

How do I keep my slimes from escaping their pens?

What are the performance impacts of 'functional' Rust?

Jazz greats knew nothing of modes. Why are they used to improvise on standards?

Can I throw a longsword at someone?

Strange behaviour of Check

Windows 10: How to Lock (not sleep) laptop on lid close?

Need a suitable toxic chemical for a murder plot in my novel

How is simplicity better than precision and clarity in prose?

What LEGO pieces have "real-world" functionality?

Using "nakedly" instead of "with nothing on"

Area of a 2D convex hull

How does the Nova's Burn power work at the 7-9 level?

Passing functions in C++

Autumning in love

Why does this iterative way of solving of equation work?

What would be Julian Assange's expected punishment, on the current English criminal law?

How to set letter above or below the symbol?

Statistical model of ligand substitution

What do I do if technical issues prevent me from filing my return on time?

What do you call the holes in a flute?

Why use gamma over alpha radiation?

When communicating altitude with a '9' in it, should it be pronounced "nine hundred" or "niner hundred"?

3 doors, three guards, one stone

Blender game recording at the wrong time



Handling SQL updates for production deployments



Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)MySQL Workbench team strategyExport .MWB to working .SQL file using command lineBest practices for deploying and updating SQL Server as part of code deploymentDACPAC and database drift: db users are always detected as drift even when nothing has changedHow to use mysql to migrate database changes from local databases?Builds and deployments to 300+ databases utilizing SSDT/VSTSWhere should I store a custom “.my.cnf” file?SSDT Drop and Recreate Tables when nothing has changedMySQL Schema Change Management Tools?Query is extremely slow after ALTER and deploying of updates





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}







4















I'm researching for quite a long time now (about two months) and haven't found a good approach for this, so I'll ask you experts out there in the hope of some enlightenment.



I run a pretty traditional LAMP web application in the cloud. The source code is VCS'ed with Git, as well as database schema (split SQL files for structure and default data).



Deploying a new environment is quite a joy, development or production. Dev environments are deployed with Vagrant plus some nice shell provisioning script I wrote. Production environments are deployed using a subset of the Vagrant provisioning script, which is also very easy to do.



When the development cycle imposes a change to the SQL schema (structure or data), things starts to break. In that case, developers changes the core SQL scripts so all the changes gets into the VCS, which is totally fine for brand new environments.



The problem sits within already deployed production environments: for those, I have to manually get all the patches for the SQL changes since the revision actually running in all production servers first so I can apply them manually after. This takes a long time to happen, is a quite fragile task and error prone.



Initially, I thought I can move the SQL changes away from the core SQL files smaller subsets, letting the core SQL files as a starting point. When there is something to change at SQL structure, I could just tell devs to create a new SQL file with only the changes from that development cycle.



So, for instance:



- structure_begin.sql: the SQL schema as it is now.
|- structure_devcycle1.sql: first set of changes to the structure
|- structure_devcycle2.sql: second set of changes to the structure, and so forth...


Then, by using Git hooks, I can selectively and automatically deploy them on production. But I don't know if this is a good approach. And here I can ask:




  1. Does somebody out there have solved this puzzle?

  2. What are the best practices for release management and deploy automation for SQL changes in production environments?

  3. Are there any open source tools that can help in this process? (I'm aware of Sqitch, but I'm not quite sure if it is the right one for the job.)










share|improve this question
















bumped to the homepage by Community 8 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
















  • PostgreSQL has a VERY cool feature for this. we can do transactional DDLs. this means that you can rollback any create table, create index, alter table, etc. this helps to avoid problems related to syntax errors and all that. but, this is not all. CREATE EXTENSION (postgresql.org/docs/9.4/static/sql-createextension.html). it allows you to version stuff. so you can directly jump from version to the other. no need to say that it is transactional as well. consider switching - mysql cannot do this kind of stuff.

    – Hans-Jürgen Schönig
    Nov 15 '14 at 9:29











  • Indeed, this is a very nice feature of PostgreSQL, @Hans-JürgenSchönig. Unfortunately, the application I manage is quite an old one, do not use a DB HAL, so just migrate it is quite a huge job for now and is not in the sight for at least the medium term (1~2y). I'm looking for a solution that could be implemented with the present stack, where MySQL/MariaDB is the main RDBMS.

    – Deny Dias
    Nov 16 '14 at 17:54













  • I would suggest reading this article on InfoQ it touches on a number of points brought up in this discussion

    – user3660241
    Nov 23 '14 at 16:10











  • That article is gold, @user3660241! Thanks for sharing it!

    – Deny Dias
    Nov 24 '14 at 20:08


















4















I'm researching for quite a long time now (about two months) and haven't found a good approach for this, so I'll ask you experts out there in the hope of some enlightenment.



I run a pretty traditional LAMP web application in the cloud. The source code is VCS'ed with Git, as well as database schema (split SQL files for structure and default data).



Deploying a new environment is quite a joy, development or production. Dev environments are deployed with Vagrant plus some nice shell provisioning script I wrote. Production environments are deployed using a subset of the Vagrant provisioning script, which is also very easy to do.



When the development cycle imposes a change to the SQL schema (structure or data), things starts to break. In that case, developers changes the core SQL scripts so all the changes gets into the VCS, which is totally fine for brand new environments.



The problem sits within already deployed production environments: for those, I have to manually get all the patches for the SQL changes since the revision actually running in all production servers first so I can apply them manually after. This takes a long time to happen, is a quite fragile task and error prone.



Initially, I thought I can move the SQL changes away from the core SQL files smaller subsets, letting the core SQL files as a starting point. When there is something to change at SQL structure, I could just tell devs to create a new SQL file with only the changes from that development cycle.



So, for instance:



- structure_begin.sql: the SQL schema as it is now.
|- structure_devcycle1.sql: first set of changes to the structure
|- structure_devcycle2.sql: second set of changes to the structure, and so forth...


Then, by using Git hooks, I can selectively and automatically deploy them on production. But I don't know if this is a good approach. And here I can ask:




  1. Does somebody out there have solved this puzzle?

  2. What are the best practices for release management and deploy automation for SQL changes in production environments?

  3. Are there any open source tools that can help in this process? (I'm aware of Sqitch, but I'm not quite sure if it is the right one for the job.)










share|improve this question
















bumped to the homepage by Community 8 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
















  • PostgreSQL has a VERY cool feature for this. we can do transactional DDLs. this means that you can rollback any create table, create index, alter table, etc. this helps to avoid problems related to syntax errors and all that. but, this is not all. CREATE EXTENSION (postgresql.org/docs/9.4/static/sql-createextension.html). it allows you to version stuff. so you can directly jump from version to the other. no need to say that it is transactional as well. consider switching - mysql cannot do this kind of stuff.

    – Hans-Jürgen Schönig
    Nov 15 '14 at 9:29











  • Indeed, this is a very nice feature of PostgreSQL, @Hans-JürgenSchönig. Unfortunately, the application I manage is quite an old one, do not use a DB HAL, so just migrate it is quite a huge job for now and is not in the sight for at least the medium term (1~2y). I'm looking for a solution that could be implemented with the present stack, where MySQL/MariaDB is the main RDBMS.

    – Deny Dias
    Nov 16 '14 at 17:54













  • I would suggest reading this article on InfoQ it touches on a number of points brought up in this discussion

    – user3660241
    Nov 23 '14 at 16:10











  • That article is gold, @user3660241! Thanks for sharing it!

    – Deny Dias
    Nov 24 '14 at 20:08














4












4








4


3






I'm researching for quite a long time now (about two months) and haven't found a good approach for this, so I'll ask you experts out there in the hope of some enlightenment.



I run a pretty traditional LAMP web application in the cloud. The source code is VCS'ed with Git, as well as database schema (split SQL files for structure and default data).



Deploying a new environment is quite a joy, development or production. Dev environments are deployed with Vagrant plus some nice shell provisioning script I wrote. Production environments are deployed using a subset of the Vagrant provisioning script, which is also very easy to do.



When the development cycle imposes a change to the SQL schema (structure or data), things starts to break. In that case, developers changes the core SQL scripts so all the changes gets into the VCS, which is totally fine for brand new environments.



The problem sits within already deployed production environments: for those, I have to manually get all the patches for the SQL changes since the revision actually running in all production servers first so I can apply them manually after. This takes a long time to happen, is a quite fragile task and error prone.



Initially, I thought I can move the SQL changes away from the core SQL files smaller subsets, letting the core SQL files as a starting point. When there is something to change at SQL structure, I could just tell devs to create a new SQL file with only the changes from that development cycle.



So, for instance:



- structure_begin.sql: the SQL schema as it is now.
|- structure_devcycle1.sql: first set of changes to the structure
|- structure_devcycle2.sql: second set of changes to the structure, and so forth...


Then, by using Git hooks, I can selectively and automatically deploy them on production. But I don't know if this is a good approach. And here I can ask:




  1. Does somebody out there have solved this puzzle?

  2. What are the best practices for release management and deploy automation for SQL changes in production environments?

  3. Are there any open source tools that can help in this process? (I'm aware of Sqitch, but I'm not quite sure if it is the right one for the job.)










share|improve this question
















I'm researching for quite a long time now (about two months) and haven't found a good approach for this, so I'll ask you experts out there in the hope of some enlightenment.



I run a pretty traditional LAMP web application in the cloud. The source code is VCS'ed with Git, as well as database schema (split SQL files for structure and default data).



Deploying a new environment is quite a joy, development or production. Dev environments are deployed with Vagrant plus some nice shell provisioning script I wrote. Production environments are deployed using a subset of the Vagrant provisioning script, which is also very easy to do.



When the development cycle imposes a change to the SQL schema (structure or data), things starts to break. In that case, developers changes the core SQL scripts so all the changes gets into the VCS, which is totally fine for brand new environments.



The problem sits within already deployed production environments: for those, I have to manually get all the patches for the SQL changes since the revision actually running in all production servers first so I can apply them manually after. This takes a long time to happen, is a quite fragile task and error prone.



Initially, I thought I can move the SQL changes away from the core SQL files smaller subsets, letting the core SQL files as a starting point. When there is something to change at SQL structure, I could just tell devs to create a new SQL file with only the changes from that development cycle.



So, for instance:



- structure_begin.sql: the SQL schema as it is now.
|- structure_devcycle1.sql: first set of changes to the structure
|- structure_devcycle2.sql: second set of changes to the structure, and so forth...


Then, by using Git hooks, I can selectively and automatically deploy them on production. But I don't know if this is a good approach. And here I can ask:




  1. Does somebody out there have solved this puzzle?

  2. What are the best practices for release management and deploy automation for SQL changes in production environments?

  3. Are there any open source tools that can help in this process? (I'm aware of Sqitch, but I'm not quite sure if it is the right one for the job.)







mysql deployment change-management






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 24 '16 at 19:35







Deny Dias

















asked Nov 14 '14 at 23:29









Deny DiasDeny Dias

213




213





bumped to the homepage by Community 8 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.







bumped to the homepage by Community 8 mins ago


This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.















  • PostgreSQL has a VERY cool feature for this. we can do transactional DDLs. this means that you can rollback any create table, create index, alter table, etc. this helps to avoid problems related to syntax errors and all that. but, this is not all. CREATE EXTENSION (postgresql.org/docs/9.4/static/sql-createextension.html). it allows you to version stuff. so you can directly jump from version to the other. no need to say that it is transactional as well. consider switching - mysql cannot do this kind of stuff.

    – Hans-Jürgen Schönig
    Nov 15 '14 at 9:29











  • Indeed, this is a very nice feature of PostgreSQL, @Hans-JürgenSchönig. Unfortunately, the application I manage is quite an old one, do not use a DB HAL, so just migrate it is quite a huge job for now and is not in the sight for at least the medium term (1~2y). I'm looking for a solution that could be implemented with the present stack, where MySQL/MariaDB is the main RDBMS.

    – Deny Dias
    Nov 16 '14 at 17:54













  • I would suggest reading this article on InfoQ it touches on a number of points brought up in this discussion

    – user3660241
    Nov 23 '14 at 16:10











  • That article is gold, @user3660241! Thanks for sharing it!

    – Deny Dias
    Nov 24 '14 at 20:08



















  • PostgreSQL has a VERY cool feature for this. we can do transactional DDLs. this means that you can rollback any create table, create index, alter table, etc. this helps to avoid problems related to syntax errors and all that. but, this is not all. CREATE EXTENSION (postgresql.org/docs/9.4/static/sql-createextension.html). it allows you to version stuff. so you can directly jump from version to the other. no need to say that it is transactional as well. consider switching - mysql cannot do this kind of stuff.

    – Hans-Jürgen Schönig
    Nov 15 '14 at 9:29











  • Indeed, this is a very nice feature of PostgreSQL, @Hans-JürgenSchönig. Unfortunately, the application I manage is quite an old one, do not use a DB HAL, so just migrate it is quite a huge job for now and is not in the sight for at least the medium term (1~2y). I'm looking for a solution that could be implemented with the present stack, where MySQL/MariaDB is the main RDBMS.

    – Deny Dias
    Nov 16 '14 at 17:54













  • I would suggest reading this article on InfoQ it touches on a number of points brought up in this discussion

    – user3660241
    Nov 23 '14 at 16:10











  • That article is gold, @user3660241! Thanks for sharing it!

    – Deny Dias
    Nov 24 '14 at 20:08

















PostgreSQL has a VERY cool feature for this. we can do transactional DDLs. this means that you can rollback any create table, create index, alter table, etc. this helps to avoid problems related to syntax errors and all that. but, this is not all. CREATE EXTENSION (postgresql.org/docs/9.4/static/sql-createextension.html). it allows you to version stuff. so you can directly jump from version to the other. no need to say that it is transactional as well. consider switching - mysql cannot do this kind of stuff.

– Hans-Jürgen Schönig
Nov 15 '14 at 9:29





PostgreSQL has a VERY cool feature for this. we can do transactional DDLs. this means that you can rollback any create table, create index, alter table, etc. this helps to avoid problems related to syntax errors and all that. but, this is not all. CREATE EXTENSION (postgresql.org/docs/9.4/static/sql-createextension.html). it allows you to version stuff. so you can directly jump from version to the other. no need to say that it is transactional as well. consider switching - mysql cannot do this kind of stuff.

– Hans-Jürgen Schönig
Nov 15 '14 at 9:29













Indeed, this is a very nice feature of PostgreSQL, @Hans-JürgenSchönig. Unfortunately, the application I manage is quite an old one, do not use a DB HAL, so just migrate it is quite a huge job for now and is not in the sight for at least the medium term (1~2y). I'm looking for a solution that could be implemented with the present stack, where MySQL/MariaDB is the main RDBMS.

– Deny Dias
Nov 16 '14 at 17:54







Indeed, this is a very nice feature of PostgreSQL, @Hans-JürgenSchönig. Unfortunately, the application I manage is quite an old one, do not use a DB HAL, so just migrate it is quite a huge job for now and is not in the sight for at least the medium term (1~2y). I'm looking for a solution that could be implemented with the present stack, where MySQL/MariaDB is the main RDBMS.

– Deny Dias
Nov 16 '14 at 17:54















I would suggest reading this article on InfoQ it touches on a number of points brought up in this discussion

– user3660241
Nov 23 '14 at 16:10





I would suggest reading this article on InfoQ it touches on a number of points brought up in this discussion

– user3660241
Nov 23 '14 at 16:10













That article is gold, @user3660241! Thanks for sharing it!

– Deny Dias
Nov 24 '14 at 20:08





That article is gold, @user3660241! Thanks for sharing it!

– Deny Dias
Nov 24 '14 at 20:08










1 Answer
1






active

oldest

votes


















0














I am not sure if you still need an answer, but I had the same problem with managing and deploying schema changes. Sqitch was the first thing I tried to mechanize migrations, but I did not really like it.



Just as you described, we have separate .sql files with CREATE TABLE and other instructions reflecting our database schema. We created simple tool to generate SQL patches when schema changes. Take a look at it: https://github.com/condograde/sqlibrist. There is detailed tutorial, how to use it.






share|improve this answer
























  • Hi, Serj! First, sqlibrist is a hell of a name! Congratulations to the guy/gal that came with this up! Yes, the original situation is now under control. But sqlibrist could definitively improve it. I'm not going to hit the accepted answer button now because I have to investigate it further, which will take a while. But for sure your answer deserves my upvote. ;)

    – Deny Dias
    Mar 24 '16 at 19:31












Your Answer








StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f82704%2fhandling-sql-updates-for-production-deployments%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









0














I am not sure if you still need an answer, but I had the same problem with managing and deploying schema changes. Sqitch was the first thing I tried to mechanize migrations, but I did not really like it.



Just as you described, we have separate .sql files with CREATE TABLE and other instructions reflecting our database schema. We created simple tool to generate SQL patches when schema changes. Take a look at it: https://github.com/condograde/sqlibrist. There is detailed tutorial, how to use it.






share|improve this answer
























  • Hi, Serj! First, sqlibrist is a hell of a name! Congratulations to the guy/gal that came with this up! Yes, the original situation is now under control. But sqlibrist could definitively improve it. I'm not going to hit the accepted answer button now because I have to investigate it further, which will take a while. But for sure your answer deserves my upvote. ;)

    – Deny Dias
    Mar 24 '16 at 19:31
















0














I am not sure if you still need an answer, but I had the same problem with managing and deploying schema changes. Sqitch was the first thing I tried to mechanize migrations, but I did not really like it.



Just as you described, we have separate .sql files with CREATE TABLE and other instructions reflecting our database schema. We created simple tool to generate SQL patches when schema changes. Take a look at it: https://github.com/condograde/sqlibrist. There is detailed tutorial, how to use it.






share|improve this answer
























  • Hi, Serj! First, sqlibrist is a hell of a name! Congratulations to the guy/gal that came with this up! Yes, the original situation is now under control. But sqlibrist could definitively improve it. I'm not going to hit the accepted answer button now because I have to investigate it further, which will take a while. But for sure your answer deserves my upvote. ;)

    – Deny Dias
    Mar 24 '16 at 19:31














0












0








0







I am not sure if you still need an answer, but I had the same problem with managing and deploying schema changes. Sqitch was the first thing I tried to mechanize migrations, but I did not really like it.



Just as you described, we have separate .sql files with CREATE TABLE and other instructions reflecting our database schema. We created simple tool to generate SQL patches when schema changes. Take a look at it: https://github.com/condograde/sqlibrist. There is detailed tutorial, how to use it.






share|improve this answer













I am not sure if you still need an answer, but I had the same problem with managing and deploying schema changes. Sqitch was the first thing I tried to mechanize migrations, but I did not really like it.



Just as you described, we have separate .sql files with CREATE TABLE and other instructions reflecting our database schema. We created simple tool to generate SQL patches when schema changes. Take a look at it: https://github.com/condograde/sqlibrist. There is detailed tutorial, how to use it.







share|improve this answer












share|improve this answer



share|improve this answer










answered Mar 23 '16 at 13:36









Serhii ZavadskyiSerhii Zavadskyi

1




1













  • Hi, Serj! First, sqlibrist is a hell of a name! Congratulations to the guy/gal that came with this up! Yes, the original situation is now under control. But sqlibrist could definitively improve it. I'm not going to hit the accepted answer button now because I have to investigate it further, which will take a while. But for sure your answer deserves my upvote. ;)

    – Deny Dias
    Mar 24 '16 at 19:31



















  • Hi, Serj! First, sqlibrist is a hell of a name! Congratulations to the guy/gal that came with this up! Yes, the original situation is now under control. But sqlibrist could definitively improve it. I'm not going to hit the accepted answer button now because I have to investigate it further, which will take a while. But for sure your answer deserves my upvote. ;)

    – Deny Dias
    Mar 24 '16 at 19:31

















Hi, Serj! First, sqlibrist is a hell of a name! Congratulations to the guy/gal that came with this up! Yes, the original situation is now under control. But sqlibrist could definitively improve it. I'm not going to hit the accepted answer button now because I have to investigate it further, which will take a while. But for sure your answer deserves my upvote. ;)

– Deny Dias
Mar 24 '16 at 19:31





Hi, Serj! First, sqlibrist is a hell of a name! Congratulations to the guy/gal that came with this up! Yes, the original situation is now under control. But sqlibrist could definitively improve it. I'm not going to hit the accepted answer button now because I have to investigate it further, which will take a while. But for sure your answer deserves my upvote. ;)

– Deny Dias
Mar 24 '16 at 19:31


















draft saved

draft discarded




















































Thanks for contributing an answer to Database Administrators Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f82704%2fhandling-sql-updates-for-production-deployments%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

ORA-01691 (unable to extend lob segment) even though my tablespace has AUTOEXTEND onORA-01692: unable to...

Always On Availability groups resolving state after failover - Remote harden of transaction...

Circunscripción electoral de Guipúzcoa Referencias Menú de navegaciónLas claves del sistema electoral en...