4d is 4th dimension

Care and feeding of your 4D database

4th Dimension's superiority over other databases is partially due to the fact that 4D requires minimal maintenance. However, all databases are complex and thus susceptible to problems unless certain maintenance procedures are followed. Here are a few guidelines to keep a database running trouble free.



4D Wizard's Home

office@awizard.com

818-824-6078

© 2006-2010 Palmer Tilden of Wizard


What can go wrong
An overview of the components of the 4th Dimension package
      Structure
      Data
      Mac4dx/Win4dx folder
      4th Dimension - single user
      4D Server - multiple users
      Index
      Compiled
Launching Your 4D Database
Backing Up
How do I know if I have data corruption?
How to handle a data problem?
Preventive Medicine
     Norton
     DiskWarrior
      4D Tools
            Check
            Repair
            Compact
Monthly Tune-up
Software Updates
Memory
Ownership

What can go wrong:

Your database is vital to your business. It must work correctly since it affects your entire company. Thus, it is necessary to do some routine maintenance on it.

Although modern computers are marvels of technology, they do occasionally develop problems. In general, these glitches are due to the multitude of complex software that is continually being run on the computers.

Here is an example how a typical problem arises with a 4D database.

Although most 4th Dimension data files start out small, it is not uncommon to see data files that have grown to 50, 100, or even 500 megabytes or more in size. Let's suppose, for a moment, that a client's data file grows to 50MB. That means there are fifty million pieces of information stored in the file. This presents a sizable record-keeping task for 4th Dimension and the client's hard disk. If 4D or the hard disk makes a single mistake, thus losing track of just one of those 50,000,000 pieces of information, data corruption results.

It may be necessary to call on any one of those 50,000,000 pieces of information any time. Hence, 4D has to know, at all times, exactly where every single piece of information is located. It must also be aware of exactly how each piece of data relates to the other 49,999,999.

Every time a new record is created, 4D has to decide where to put it and how to keep track of it. Each time a piece of information is changed, 4D has to first locate the original data. 4D then has to delete it and replace it with the updated version. If the new information takes up more space than the old, 4D has to find a new place to store it. This data-shuffling exercise takes place thousands of times a day in most 4D databases.

What happens if the client's computer crashes, as do all computers occasionally? If 4D is saving data onto the hard disk when the crash occurs, then there is a chance of that data becoming corrupted. If only part of a record is saved before the crash, then some of the valuable data is lost.

As will be shown later in this document, there are precautionary measures which can help greatly to avoid the aforementioned mishaps. (See Preventative Medicine below.)

If, in spite of all precautions, a mishap does damage the client's database, then proper backup procedures will allow the client to escape virtually unscathed from such an unfortunate occurrence. (See Backing Up below.)

Now that we have briefly examined some possible problems that can plague a user of 4th Dimension, let's take a look at 4D's crucial parts.

An overview of the components of the 4th Dimension package:

There are 3, 4 or 5 components of a Macintosh 4D database; on a PC there are more. Here the main parts of a 4D database.

Naming conventions:

The screens, menus, buttons, data entry areas, and all the programming code are in one file which is usually named by either the client or the 4D programmer. In the following example it will arbitrarily be called "MyDatabase". (On a PC it is two files: "MyDatabase.4DB" and "MyDatabase.RSR". A compiled version is named "MyDatabase.4DC" and "MyDatabase.RSR")

The 4D programmer refers to this file as the database "structure," since it is the structure of the database design. It is where the programmer does all his work. There may be a version number added to the end of the name. This version number will increment whenever an updated version of the database is distributed. The client should always use the latest version of the "MyDatabase" file.

The data file is where everything that is entered into the 4D database is kept. This second file usually bears the same name as the structure followed by the suffix ".data." Therefore, in this example, it will be called "MyDatabase.data." (On a PC or newer versions of 4D it is two files: "MyDatabase.4DD" and "MyDatabase.4DR".) The "MyDatabase.data" is what should be backed up because it is usually the only thing that changes each day as additional records (data) are entered.

Sometimes 4D cannot do everything that the programmer believes is required of a specific database. In such instances a plug-in must be purchased by the programmer to give the database more functionality. The plug-ins are stored in a folder named "MAC4DX" or Plugins (on a PC it is named "WIN4DX"). If this folder is required, it is found in the same outer folder as MyDatabase and MyDatabase.data. The client should never remove anything from the MAC4DX (WIN4DX) folder.

The engine of the database is the 4th Dimension Application software. It is a separate file that can be named a variety of things such as 4th Dimension, 4D, 4D Runtime Classic, or 4D Server. Usually it has a version number after it. If 4D is being run on multiple computers, then 4D Server is software on a dedicated computer called a Server. 4D Client can then be found on each workstation computer.

It is strongly recommended that when the 4D Server is used that it be run on a Dedicated Server (one that has no other programs running on it.) Wizard has noted far greater instances of data corruption in offices that run 4D on non-dedicated Servers. A non-dedicated Server is one that is serving double-duty. This means that it is being simultaneously employed as a user's workstation and as the server for the database.

Wizard can offer one theory as to why data corruption may be more prevalent in non-dedicated server environments. Wizard believes that when the user with a non-dedicated server happens to have a system crash while simultaneously running some other program, the crash will bring down the database as well. If the database happened to have been saving a record or updating an index at the precise moment of the crash, then that record or index will become corrupted.

If a 4D Server is being used, it is suggested that the client should not use a screen-saver on the Server computer. A screen-saver could interfere with the operation of 4D Server. It would be better to just turn off the monitor or dim the screen.

If 4D Server is being used on a Mac, then File Sharing should be turned off. This will speed up the functioning of the CPU and the running of 4th Dimension.

An Index is the way that any database marks certain fields so that they can be searched or sorted faster. A non-indexed "Last Name Field" may add up to several minutes to the search, while an indexed one would take less than a second to be found. In 4D, the indexed field is in bold type within the 4D Query Editor. If the client finds that searches are often made on a non-indexed field, then the programmer should be asked to index it. This process only takes a moment.

A compiled database runs faster than an uncompiled one. The client's 4th Dimension programmer may run a software compiler on the "MyDatabase" structure to change the 4D code to machine language so that the client's program will run much faster. If the MyDatabase structure does not have a custom icon on the file, a look at the icon of MyDatabase structure will reveal if it is compiled. A little running man on the 4D icon signifies that the structure is compiled. (On a PC a compiled version is named "MyDatabase.4DC" and "MyDatabase.RSR")

One drawback to compiling the database is that the programmer needs the uncompiled master to make any changes to the database. Please note that no one can get into the code of a compiled database. The 4D programmer always keeps the uncompiled master in order to make future changes. See Ownership below.

Launching Your Database:

To start running the database one should always drag-and-drop the MyDatabase file on top of 4D. Alternatively one can double-click on the MyDatabase file. One should not drag-and-drop or double-click on the MyDatabase.data file. This method will work, but sometimes an error message is generated.

If 4D Server is used, then at the individual workstations, the database is started by double-clicking the 4D Client file.

Backing Up:

Please note that whenever the database is to be backed up, the MyDatabase.data file is the file to use for this purpose.

Here are Wizard's Backup Software Recommendations:

Although there are a few backup utilities on the market, one stands out as a clear winner. This is Retrospect, by Dantz www.dantz.com. This program has become the standard to which all others are compared. Retrospect has the ability to completely automate your backup procedures, running them unattended at night while you are away. In addition, it has built-in compression and decompression capabilities. Retrospect has excellent archiving facilities, allowing you to quickly locate and recover old archives to the destination of your choice.

It should be noted that Retrospect is slightly complex. More than a quick glance at the instructions will be needed to take advantage of Retrospect's various attributes. This is a small price to pay for the ease of use and robust capabilities featured in this backup utility.

Wizard's Backup Media Recommendation:

Every backup media alternative has its own particular pros and cons. In certain situations, each has a place. The backup setup that Wizard prefers and recommends for 4D databases is a combination of the Hard Disk option and the Removable Cartridge option.

One should back up to a separate hard disk that can be activated immediately if the primary hard disk goes down. Backup up to the second hard disk should occur at least once a day. By following this procedure, the client will never lose more than one day's data, should a mishap occur.

Retrospect should be used to back up to a Removable Cartridge Drive. At least 17 removable cartridges should be used for this purpose, one for each daily backup and one for each monthly backup. The best Removable Cartridge Drive option is a DAT drive with DAT tapes. The tapes cost only $10 each.

If the data is small enough, then the most inexpensive way to back up is to use a Zip drive. These come in 100MB and 250MB capacities. The client can just drag-and drop the MyDatabase.data file onto the zip disk each day.

The organization of backups:

In the world of computers there are three classes of backup:

  1. On-site backup of changed data.
  2. On-site full backup.
  3. Off-site full backup.

Three levels of hard disk backup should be maintained:

  1. A backup of the data that has changed since your last full backup. This is kept on a secondary hard disk and or DAT tape drive.
  2. A full backup on a tape within the client's office.
  3. A full backup on a tape outside the office. This is necessary in case of fire or theft.

Incremental Backups should be maintained. A 4D database is dynamic. For that reason, incremental backups are required. Incremental backups are full backups that reflect the status of the data at a specified point in time.

Here are two good uses that can be made of incremental backups:

  1. Record Recovery. There are certain records in the client's database that are more or less permanent. Examples of these would be Customer records, or Inventory Item records. If any of those permanent records are lost, incremental backups permit the client to go back to an old copy of the database and retrieve those records.
  2. A Glimpse of History.
    Once in awhile, the client will need to know what the status of the database was at a given date. If incremental backups have been maintained, the client can go to the backup which is closest to that date and get a pretty good idea of the database's status at that point in time.
Here is the method for keeping backups as glimpses of history:
  • Make daily incremental backups, keep them for a week.
  • Make monthly incremental backups, keep them for a year.

Managing Your Incremental Backups:

Here is a well-established technique used for managing incremental backups.

For Daily backups, create five disks or tapes labeled Monday, Tuesday, Wednesday, Thursday, and Friday. At the end of the day on Monday, use the Monday disk to make the new Monday backup; do the same on Tuesday, and so on.

For Monthly backups, create twelve disks or tapes labeled January, February, March, etc. After you have done the end-of-month posting and printed the month's hard copy reports at the end of January, use the January disk to make the new January backup. Then do the same at the end of February, March, and so on.

WARNING: It is possible to go for long periods of time with no mishaps occurring to a client's 4D database. This can cause an inexperienced client to periodically skip the recommended backups.

Eventually a mishap will occur to the database. If this happens when the proper backups have not been made, the client's operation will suffer a calamitous blow. In such cases the client realizes, all too late, that maintaining a rigorous backup procedure is a small price to pay for preventing a tragic setback to the client's business.

WARNING #2: Wizard knows of many cases where a client's 4D data got saved on top of the prior backup. Eventually, such a client unknowingly developed a major glitch in the database. Before the problem had been discovered, the prior sound backup had been overwritten with the current faulty one.

Thus, it behooves the client to keep a sequence of backups. By doing so, the client is well covered when a problem in the database is discovered, for then there will most certainly be a prior backup which has uncorrupted data on it.

Wizard was once called in to help a large company recover its data. Since the company had made a daily backup, the recovery exercise appeared to be a simple one. Then the bad news was discovered. The company had backed up the 4D Application every day and had never once backed up the data. Since hundreds of invoices were normally entered into the database each day, it cost the company 6 months of time and over $10,000 to reenter the lost data.

How does a client discover data corruption in the company's 4D database?

  1. An error message may appear, saying something about a bad record or index.

  2. If a simple, single-level sort or search is performed and the results are incorrect, that is probably a sign of a damaged index or a "gremlin."

    A "gremlin" is an invisible control character created when the client intended to use the shift key but hit the control key instead. Since the resulting control character is usually invisible, the user is not likely to catch the typing error. Thus, the control character is stored in the database. In the 4th Dimension community, it has been very well documented that these control characters can cause data corruption. If you do a simple search for "John" you may not find all the Johns because one of the "john records" contained 5 characters instead of 4, where the fifth character was the "gremlin."

    Another way gremlins can develop is in the process of Importing data. Some databases, spreadsheets, and word processors may use those control characters to store formatting information. A client, who is unaware of this problem, might import data without stripping out the control characters.

    To avoid this problem, the client will need the help of the 4D developer. The 4D programmer can add code which is used to strip control characters from any alpha or text field. If the client starts to experience inexplicable data corruption problems, especially after an import, it would be wise to spend a little extra time and money by having the 4D programmer add some code which will strip out the gremlins. If Wizard created the database then the client need not worry, since Wizard always adds this gremlin-stripping feature to new databases.

  3. If the client begins experiencing frequent 4D freezes and bombs , that's a sure sign that there is, or soon will be, damage in the 4D database.

  4. Another cause of crashing could be that the 4D database has not been programmed correctly. In such a case the client should try to contact the 4D programmer.

    If a 4D programmer has not been recently working on the database, then it is not a bug; it more likely that the client has a data problem.

  5. The client's database has been working well for a while, but suddenly does not perform all of its designated functions.

  6. There is always a chance that someone on the client's network will pick up a computer virus . In such cases, the virus can quickly spread throughout the network and corrupt the client's data. A Mac user should consider a virus to be a problem's source only as a last resort, since Macintosh viruses are rare. On the other hand, PC viruses circulate frequently.

How to handle a data problem?

First run 4D Tools. See the next section.

Second, contact your 4D programmer.

Lastly, contact Wizard. Solving database problems is a Wizard specialty.

Preventive Medicine:

The most dreaded phrase in the computer language is "hard disk crash." Just what do those words mean, and why are they so feared?

When a hard disk malfunctions to the point that it is no longer able to accurately read, write, or locate files, it has crashed. Interestingly enough, a hard disk crash usually does not mean that the information on the hard disk has been totally destroyed; it normally means that just a few pieces of information on the hard disk have been lost. These pieces of information are collectively called the directory. When the directory becomes corrupted, the hard disk cannot find files, even though the files are still there!

Note that this directory problem has its counterpart in another aspect of every day living. If one wants to call a business associate on the phone, the phone number must first be obtained from a directory. That directory might be stored in one's head, but it is still a directory. If the number is forgotten, then the associate can't be called.

Norton Disk Doctor and DiskWarrior:

Norton Disk Doctor: For years Norton Disk Doctor was once the standard in the computer industry for repairing hard disk problems. It is no longer recommended.

DiskWarrior (for Mac only): A study of professional computer users found that DiskWarrior www.alsoft.com is now considered to be the best program for maintaining the integrity of a hard disk. DiskWarrior fixes and optimizes the hard disk directory, thus preventing files from becoming corrupted or lost.

Wizard suggests that you own DiskWarrior. Wizard runs DiskWarrior every month weeks DiskWarrior to fix things before they become problems.

4D Tools is a free utility from 4D Inc. that prevents data problems. 4D tools also fixes some database corruption.

When to use 4D Tools:

Occasionally, when several hundred or more records are deleted, 4D will need to repair itself. Even if no symptoms of corruption are noted, it is a good idea to run 4D Tools after deleting a large number of records.

Corruption will sometimes be noted after thousands of records are imported into a database. Even when no symptoms of corruption are noted, it is still a good idea to run 4D Tools after importing a large number of records.

4D Tools should be run on any of the following occasions:

Even if none of the above symptoms occurs, 4D Tools should still be run once a month or so.

Running 4D Tools is a relatively simple operation. The program itself does all the work; all it requires is that a couple of buttons be pushed.

Please note that before running 4D Tools, it should be determined that there is enough space on the hard disk to duplicate the entire data file.

Every time 4D Tools is utilized, the "Check Only and Create a log" option should be run first. In the newer versions of 4D this function is started by clicking a "Check All" button. This causes the database and indexes to be checked. Any damage to specific records and indexes is reported. If no damage is found, 4D Tools issues a message stating "Everything seems OK." This is appropriate because the Check Routine is not able to detect certain types of data corruption. Therefore, sometimes everything will appear to be OK when such is not the case. This exception occurs infrequently.

4D Tools creates a damage log named "Journal File" which is a text file that can be opened with TextEdit, WordPad, MS Word, or some other word processor. The log cannot be read from within 4D Tools. If there are damage entries in the "Journal File," then the "Check and Fix" option should be run. In the newer versions of 4D Tools this option has been changed to "Repair All."

When the Check and Fix (Repair All) function is run, 4D Tools goes through the same process and creates the same log that it did in the Check-only option. The difference is that Check and Fix (Repair All) attempts to fix the damage where Check-only just reports its presence. The repair is not always successfully accomplished. Such a failure becomes obvious when the database is relaunched, only to have the same problems still exist.

The "Recover by Tags" option should be used when the list of corrupted records is quite long, since that means the damage is extensive. Usually 4D Tools will recommend when to run the Recover by Tags option. Recover by Tags will attempt to fix any other records that are damaged. It will also try to bring back the records that have been automatically deleted due to the fact that they were damaged. The drawback is that Recover by Tags will also resurrect some records that were deleted intentionally by the user. In other words, when Recover by Tags is run, 4D Tools will recover deleted records that have not already been overwritten with new data.

Compacting Your 4D Database. When it Compacts a database, 4D Tools creates a brand new data file and compacts all the blank spaces that have been left by deleted records. This usually results in a significantly smaller data file. Before running 4D Tools Compact, the client should first confirm that there is enough space on the hard disk to duplicate the data file.

A database should be compacted following a successful running of 4D Tools "Check" or "Check and Fix" (Repair). There is no need to compact after running "Recover by Tags."

Compacting duplicates an uncompiled structure and the data file. By default, Compact renames them by placing a '2' after the file name. Of course, the client should save the old MyDatabase and MyDatabase.data until it has been confirmed that the new MyDatabase2 and MyDatabase2.data are working properly.

What should a client do if these measures do not work and the data is still bad?

The answer to this question is Contact Wizard. Wizard is highly accomplished in helping clients who have database problems.

A Monthly Tune-up should always be run:

A monthly tune-up should always be run to keep the client's hard disk and 4D database operating at peak efficiency. Following this procedure will also reduce the likelihood of data loss and corruption.

Here is what should be done each month:

After the previous steps have been taken, a screen saver may be turned back on.

Normally, you should do a Tune-up at the beginning of a new month.

The client should make it a habit to back up the database and purge old data after running the end-of-month close-out procedures (print statements, print hard copy reports, archive data, etc.), Then a tune-up should be run.

Software Updates:

Upgrades of 4th Dimension:

4D Inc. produces regular product upgrades of 4th Dimension. 4D Product Upgrades. It is wise for the client to follow their 4D programmers advice about which version of 4D to use. The programmer should be recording what bugs were fixed and what problems were encountered right after making a 4D upgrade. If the client does not have a 4D programmer, then 4D upgrades should be approached with great caution. It is probably best to wait a while before implementing a newly issued upgrade. This gives 4D Inc. time to find and fix all the bugs that inevitably will appear in any newly launched software.

In general, it should be OK for the client to update to a newer version of 4D if it has the same main version number as the one that is currently being used. For example, if the client is running 4D v2004 then it would be OK to upgrade to 4D 2004.5.x or v2004.7.x. On the other hand, if the client is using 4D v6 or v2003 then a 4D programmer's assistance should be used to make the upgrade to v6.x.

There is one exception to the above rules. Wizard has seen some cases where the data has become damaged when upgrading from 4D v6.0 to 4D v6.7. or v6.8.

Updates of the client's database by a 4D programmer:

During the programming phase, the 4D programmer will be installing or giving the client regular updates of the "MyDatabase" structure. The client should always use the latest version of the "MyDatabase" file. It is wise to keep a backup of the previous structure as a precaution. This is due to the fact that on rare occasions the programmer might make a faulty update of the database.

Memory:

On a Macintosh computer, 4D's memory requirements need to be set. The PC does most of this automatically. In either case, there are numerous steps that a 4D programmer can take to manage memory more efficiently. The most cost-effective method is for the client to buy more RAM for all of the company's computers using 4D. There is no single solution that is easier and more reliable. It should be noted that RAM prices are currently extremely low.

If an upgrade is made to a new version of 4D or if "Out of Memory" error messages are encountered, then it is advisable to increase the usable memory. This is done on a Macintosh by first "selecting" 4D when it is not running. Then one should go to the File Menu item, "Get Info" which opens a small window. Within this window, the client can increase the "Preferred Size" of memory for 4D. On a PC the client may need to have a 4D programmer change the memory settings inside the programming code.

Ownership:

The most important thing that can be done to insure the longevity of the client's 4D database is to make sure that the master copy of the "MyDatabase" structure has been turned over to the client. Without it, any fixes or enhancements to the database cannot be made should the client be unable to contact the original 4D programmer.

One of Wizard's specialties is repairing and enhancing databases created by other 4D programmers. While performing this function, Wizard has periodically been faced with calamitous situations where the original programmer has changed careers, moved, or is otherwise gone without leaving the client the master copy of the "MyDatabase" structure.

If changes cannot be made to the database as the company grows, the client will become trapped in an untenable situation.

Some companies have had to spend large sums of money creating a brand new database because the old one could not be modified. For more information on how to tell which is the uncompiled structure of a database, go to "An overview of the components of the 4th Dimension package" above.

In summary, a client may very well want future additions to the company's database. Therefore a copy of the uncompiled master of the 4D database structure must be obtained from the 4D programmer. If passwords are used, the top level password must also be turned over to the client. The client should settle for nothing less than the aforementioned items.


If you need help from a professional 4D programmer, please contact Wizard. Wizard is a first class 4D expert, specializing in enhancements or repairs of current 4th Dimension databases. Wizard always strives to satisfy customers! Wizard has the experience, flexibility and enthusiasm to do the job in the exact manner specified by the client. If your 4th Dimension database needs just a couple of changes or requires a complete overhaul, Wizard has the professionals you can count on to do the job in a most professional manner.

Top of page

4D Wizard's Home

office@awizard.com