MySQL CREATE INDEX slowing downDoes MySQL still handle indexes in this way?How can I optimize my mysql setup...
In Star Trek IV, why did the Bounty go back to a time when whales are already rare?
A Permanent Norse Presence in America
Greco-Roman egalitarianism
Fly on a jet pack vs fly with a jet pack?
Why did the EU agree to delay the Brexit deadline?
List of people who lose a child in תנ"ך
How do ground effect vehicles perform turns?
Some numbers are more equivalent than others
Is it possible to have a strip of cold climate in the middle of a planet?
What does the Rambam mean when he says that the planets have souls?
THT: What is a squared annular “ring”?
What does this horizontal bar at the first measure mean?
Diode in opposite direction?
Java - What do constructor type arguments mean when placed *before* the type?
Have I saved too much for retirement so far?
How will losing mobility of one hand affect my career as a programmer?
What (else) happened July 1st 1858 in London?
Divine apple island
Do Legal Documents Require Signing In Standard Pen Colors?
Why has "pence" been used in this sentence, not "pences"?
Transformation of random variables and joint distributions
My friend sent me a screenshot of a transaction hash, but when I search for it I find divergent data. What happened?
What is the grammatical term for “‑ed” words like these?
What is this type of notehead called?
MySQL CREATE INDEX slowing down
Does MySQL still handle indexes in this way?How can I optimize my mysql setup to create my index faster?Less RAM than Index_length MyISAMHow to speed up a loading data into MySQL from a zipped archiveCompound Index on 43 Million PostgreSQL tableWhy would the size of MySQL MyISAM table indexes (aka MYI file) not match after mysqldump import?MySQL instance stalling “doing SYNC index”Invision Power Board 3.4.6 - mysql locks during query with pending updatesHow to update 10 million+ rows in MySQL single table as Fast as possible?Order by index unexpectedly slowing query downHow to create some GIN index concurrently in Postgresql
My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with
key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.
Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:
- first index - 40 minutes
- second index - 50 minutes
- ...
- fifth index - 2 hours
- eleventh index - 3 1/2 hours
Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?
mysql performance index myisam
bumped to the homepage by Community♦ 7 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with
key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.
Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:
- first index - 40 minutes
- second index - 50 minutes
- ...
- fifth index - 2 hours
- eleventh index - 3 1/2 hours
Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?
mysql performance index myisam
bumped to the homepage by Community♦ 7 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41
add a comment |
My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with
key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.
Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:
- first index - 40 minutes
- second index - 50 minutes
- ...
- fifth index - 2 hours
- eleventh index - 3 1/2 hours
Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?
mysql performance index myisam
My analytics table has 190M rows, about 150GB ins size.
Storage engine is MyISAM with
key-buffer-size=16G
myisam_sort_buffer_size=2G
myisam-max-sort-file-size=200G.
Machine has 32GB memory.
While creating 10 simple indexes (no covering, no prefix, just column names using BTREE) I noticed:
- first index - 40 minutes
- second index - 50 minutes
- ...
- fifth index - 2 hours
- eleventh index - 3 1/2 hours
Each index adds between 1GB and 3GB to the .myi file.
Has anybody experienced a similar increase in run-time? Is there any way to get the performance of the first index creation for the other indexes?
mysql performance index myisam
mysql performance index myisam
edited Jun 14 '16 at 15:01
RolandoMySQLDBA
143k24226383
143k24226383
asked Jun 14 '16 at 14:27
Wolf MetznerWolf Metzner
61
61
bumped to the homepage by Community♦ 7 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 7 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41
add a comment |
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41
add a comment |
2 Answers
2
active
oldest
votes
You must be doing something odd with your DDL or you have a very, very old version of MySQL.
MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.
Oct 10, 2006
: Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)
May 12, 2011
: Does MySQL still handle indexes in this way? (where I mention this again)
Without any additional details from you, I can only guess. Here it goes:
I suspect your must be just doing the following:
ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);
I would like to suggest the following
SUGGESTION #1 : Build all 11 indexes at the same time
You should create all 11 indexes in a single DDL statement
ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
SUGGESTION #2 : Delay indexes building
You can disable the building of non-unique indexes so that sorting and be done.
Doing so can make a smaller BTREE for each index
ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;
SUGGESTION #3 : Use a temp table
If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.
Here is a sample of what I just described to you
CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;
GIVE IT A TRY !!!
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
add a comment |
Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:
CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...
and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use
ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...
in the future.
Thanks! Problem solved .. question answered!
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f141192%2fmysql-create-index-slowing-down%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
You must be doing something odd with your DDL or you have a very, very old version of MySQL.
MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.
Oct 10, 2006
: Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)
May 12, 2011
: Does MySQL still handle indexes in this way? (where I mention this again)
Without any additional details from you, I can only guess. Here it goes:
I suspect your must be just doing the following:
ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);
I would like to suggest the following
SUGGESTION #1 : Build all 11 indexes at the same time
You should create all 11 indexes in a single DDL statement
ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
SUGGESTION #2 : Delay indexes building
You can disable the building of non-unique indexes so that sorting and be done.
Doing so can make a smaller BTREE for each index
ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;
SUGGESTION #3 : Use a temp table
If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.
Here is a sample of what I just described to you
CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;
GIVE IT A TRY !!!
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
add a comment |
You must be doing something odd with your DDL or you have a very, very old version of MySQL.
MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.
Oct 10, 2006
: Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)
May 12, 2011
: Does MySQL still handle indexes in this way? (where I mention this again)
Without any additional details from you, I can only guess. Here it goes:
I suspect your must be just doing the following:
ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);
I would like to suggest the following
SUGGESTION #1 : Build all 11 indexes at the same time
You should create all 11 indexes in a single DDL statement
ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
SUGGESTION #2 : Delay indexes building
You can disable the building of non-unique indexes so that sorting and be done.
Doing so can make a smaller BTREE for each index
ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;
SUGGESTION #3 : Use a temp table
If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.
Here is a sample of what I just described to you
CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;
GIVE IT A TRY !!!
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
add a comment |
You must be doing something odd with your DDL or you have a very, very old version of MySQL.
MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.
Oct 10, 2006
: Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)
May 12, 2011
: Does MySQL still handle indexes in this way? (where I mention this again)
Without any additional details from you, I can only guess. Here it goes:
I suspect your must be just doing the following:
ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);
I would like to suggest the following
SUGGESTION #1 : Build all 11 indexes at the same time
You should create all 11 indexes in a single DDL statement
ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
SUGGESTION #2 : Delay indexes building
You can disable the building of non-unique indexes so that sorting and be done.
Doing so can make a smaller BTREE for each index
ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;
SUGGESTION #3 : Use a temp table
If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.
Here is a sample of what I just described to you
CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;
GIVE IT A TRY !!!
You must be doing something odd with your DDL or you have a very, very old version of MySQL.
MyISAM Index creation can be very lethargic. I wrote about this almost ten years ago.
Oct 10, 2006
: Why does mysql drop index very very slow in a large table? (from the MySQL General Discussion Forum)
May 12, 2011
: Does MySQL still handle indexes in this way? (where I mention this again)
Without any additional details from you, I can only guess. Here it goes:
I suspect your must be just doing the following:
ALTER TABLE mytable ADD INDEX (column01);
ALTER TABLE mytable ADD INDEX (column02);
ALTER TABLE mytable ADD INDEX (column03);
...
ALTER TABLE mytable ADD INDEX (column10);
ALTER TABLE mytable ADD INDEX (column11);
I would like to suggest the following
SUGGESTION #1 : Build all 11 indexes at the same time
You should create all 11 indexes in a single DDL statement
ALTER TABLE myisamtable
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
SUGGESTION #2 : Delay indexes building
You can disable the building of non-unique indexes so that sorting and be done.
Doing so can make a smaller BTREE for each index
ALTER TABLE mytable DISABLE KEYS;
...
ALTER TABLE mytable ENABLE KEYS;
SUGGESTION #3 : Use a temp table
If you wish to test this manually, you can make a temp table, add all the indexes to the empty temp table, disable indexes, insert the data, enable indexes, and switch table names.
Here is a sample of what I just described to you
CREATE TABLE mytable_new LIKE mytable;
ALTER TABLE mytable_new
ADD INDEX (column01),
ADD INDEX (column02),
ADD INDEX (column03),
...
ADD INDEX (column10),
ADD INDEX (column11)
;
ALTER TABLE mytable_new DISABLE KEYS;
INSERT INTO mytable_new SELECT * FROM mytable;
ALTER TABLE mytable_new ENABLE KEYS;
ALTER TABLE mytable RENAME mytable_old;
ALTER TABLE mytable_new RENAME mytable;
DROP TABLE mytable_old;
GIVE IT A TRY !!!
edited Apr 13 '17 at 12:42
Community♦
1
1
answered Jun 14 '16 at 15:00
RolandoMySQLDBARolandoMySQLDBA
143k24226383
143k24226383
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
add a comment |
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
If this answer helped, please mark it accepted by clicking on the check mark in the upper left corner of my answer.
– RolandoMySQLDBA
Jun 15 '16 at 15:41
add a comment |
Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:
CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...
and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use
ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...
in the future.
Thanks! Problem solved .. question answered!
add a comment |
Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:
CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...
and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use
ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...
in the future.
Thanks! Problem solved .. question answered!
add a comment |
Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:
CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...
and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use
ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...
in the future.
Thanks! Problem solved .. question answered!
Rolando,
Thanks a lot for your explanation. I followed disc size and file access during this ordeal (46 hours, completed this morning) and yes .. even in MySql 5.6 this is what happens.
My statements were:
CREATE INDEX idx1 ....;
CREATE INDEX idx2 ....;
CREATE INDEX idx3 ....;
...
and I saw in the "show processlist" how all the data was first copied to a temp table and then indexed. Will use
ALTER TABLE ADD INDEX, ADD INDEX, ADD INDEX ...
in the future.
Thanks! Problem solved .. question answered!
answered Jun 15 '16 at 15:38
Wolf MetznerWolf Metzner
61
61
add a comment |
add a comment |
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f141192%2fmysql-create-index-slowing-down%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Unless everything use the exact same column structure , otherwise you cannot correlate them. You can sample part of the data into another machine for proof of concept.
– mootmoot
Jun 14 '16 at 14:41