Dear Reader,

Here is my best shot at collating and summarizing various messages
by Denys.  Note that I did NOT edit anything that follows;  and I have
tried to be careful not to leave out other stuff that would change
the context.

Thanks,

Ken Sletten
Chairman, SIGIMAGE
#####################################################


==================================
$$$$$$$      Denys Beauchemin        $$$$$$$
$$$$$$    1998-12-29 initial proposal    $$$$$$
==================================

> I concur this enchilada thing needs to be very simple.  In that vein, I
> posit the following: I would not have IMAGE manage the extra dataset.
> Rather, I would leave it as a privileged flat file, perhaps in POSIX space
> with a name like root.000 where root is the name of the database.  This
> flat file would have a very basic sort mechanism associated with it, so
> that when entries are added or deleted, the list is automatically sorted,
> in one direction only.  With judicious use of TAGs, the retrieval of the
> information would be quite rapid.  I would make the management of this
> information available through a series of DBINFO calls.  More on this
> later.
> 
> Here are some reasons why I consider doing it this way:
> 
> 1- It is much simpler to implement, as there would be no messing around
> with the existing intrinsics to coax them into reading and writing to a
> semi-transparent dataset.
> 
> 2- It would not impact any existing tools with, again, phantom datasets
> with strange names for which have to account.
> 
> 3- There would be no extra overhead for the regular IMAGE users as there
> should not be a single added line of code in the current intrinsics, save
> for the ones who would use DBINFO calls to manage the information in the
> $EXTRA dataset.
> 
> 4- The number of entries in this dataset is probably going to be very
> small, and as such, I am reticent in bringing to bear the entire power of
> IMAGE to manage these few entries.  It seems a bit, OK a lot, of a waste.
> 
> 5- The number of times this information would be accessed, compared to the
> data contained in other datasets would be very small.  It is much more
> akin to DBINFO calls retrieving information about the database or various
> datasets once during the life of the program.
> 
> I like the format of the information proposed by Steve and keeping it
> standard throughout database would be of significant importance.  As a
> backup tool vendor, it would be easy for me to find the .000 file and read
> the entries containing the POSIX names of all files associated with this
> database.  I prefer this versus DBOPENing the database and doing loops on
> DBFIND/DBGET or DBINFO calls.
> 
> Now for these DBINFO calls.  We have added several new modes to DBINFO
> over the last few years, and guess what?  As far as I remember, no DBINFO
> call created any type of critical situation in existing or even new
> databases when they were implemented.  Someone has to code to use these
> new calls.  The existing applications are totally oblivious to them,
> unlike the situation which would caused by having to noodle around with
> DGET, DBFIND, DBPUT, DBDELETE, DBUPDATE and others.
> 
> The code would only be invoked by using specific DBINFO calls.  Clean,
> unobtrusive. Safe!
> 
> 
==================================
$$$$$$$$   1998-12-31 addendum   $$$$$$$$
==================================

Consider an IMAGE database to be the equivalent of a campus.  You have the
admin building, where is housed the information about the various buildings
(datasets) in the campus, and the various capacities of said buildings and
the rooms within and how they are setup (fields and items definitions).  The
information about the relationships between the various buildings is also
kept in the admin building.  If you want to find out about anything relating
to the buildings, and the rooms in the building, you make a request using a
special form (DBINFO) after entering the admin building and you get the
information, provided you are allowed to have it.

Continuing the campus metaphor, the management of the people housed in the
rooms is done via a mechanism that uses information from the admin building
(the root file) and the small information office in each building (the
dataset file label).  Nobody gets to go in or out or changed without
checking with them.  This is excellent because using this mechanism, you can
find the exact person or persons you need in the blink of an eye.  You can
also direct a person to go to the exact room where you know you will find
them immediately when you need them again.  You do not have to worry about
this, the campus management takes care of everything for you.

This has been going swimmingly for a long time now, however there are some
chinks in the armor, if you will permit me to mix my metaphors.  When you
needed to enlarge or reduce the size of a building, you had to get everybody
out of the campus, bomb the campus into oblivion, recreate the campus with
bigger or smaller buildings and file everybody back in, according to your
registration mechanism.  If you wanted to change the number of rooms
(fields) in a building (dataset), or simply redecorate a room in a building,
you had to go through the same steps.  So a brave third party building
contractor came up with the brilliant idea that one only need to destroy and
recreate the building that needed to be changed.  He also was smart enough
to evacuate, in proper order, the occupants of the building before the
destruction and file them back inside using the same order as before.  If
the room was redecorated, he would make the occupants "adjust" to their new
surroundings.  When the construction was finished, he would inform the admin
building of the changes so they could keep track of the campus as before,
but with the new setup.

Over time, other contractors came in using the same principles, to make
changes to the campus buildings, without having to completely raze the
campus and recreate it.  Also, some map contractors were able to create
large detailed maps (b-trees) to enable mass searches of the building in one
fell swoop.  (Find all occupants who wear blue shirts.)  These maps are
housed in various ways.  Some methods use one large map warehouse, others
use little buildings underground (POSIX space).

Finally it was realized that there were too many building contractors and
map coordinators who could trip over each other.  Not only that, but the
people who took detailed pictures of the buildings and the occupants for
safekeeping (the backup tools) didn't always know which buildings were part
of the campus.  After all, the campus is in a large city (file system) which
recently added a large underground complex (POSIX space).  So it was decided
that a map detailing the various buildings and their location (POSIX file
name) was needed.  It was also felt that this map would also have a small
legend, not replacing anything in the admin building, just a little bit of
information that the admin building didn't have.

So one city planner (Steve), suggested that a building be created, just like
the other buildings in the campus, but that this building would not be seen
by anyone entering the campus.  However, the admin building would know its
existence and would provide access to the people in that building, using the
same mechanism as when managing the people in the other buildings.

A picture taker (backup vendor, Denys), suggested instead to place the map
outside the admin building, possibly in the underground area (POSIX space.)
He figured the information on the map would only be of value to the building
contractors (3rd party tools), the picture takers and a few others.  He
wanted to keep the same information as the city planner did, but just make
it so he could find it, without entering any buildings.  He also suggested
that the admin building know about the map, and would provide the
information contained on the map to anyone with a proper request (DBINFO).
The admin building would dispatch someone to create the map, or take down
the map when the campus administrator decreed it (the creator using
DBCONTROL 20 or 21.)

A city architect in the city planner's office said that was a bad idea.  He
was working on something much better.  The next day, he came back and said
that we should build the map on top of the administration building.  The
only problem is that the admin building is very, very full.  He would get a
contractor to destroy and rebuild the admin building and have the map there.
If the picture taker wanted to get the information on the map, he could
bloody well enter the building, climb up the stairs and poke around the
rooms until he found what he wanted.  Or he could make a request at the
admin building and the information would be given to him.  The picture taker
didn't want to intrude or disturb anyone.  Usually, he came in the dead of
night when the campus was closed and he didn't want to wake up the whole
campus just to know where all the buildings were.  Since he came by subway
anyways, he would just as soon stop off at the underground map and know
instantly where everything was.  Then he could go around to all the
buildings and take his pictures.  He could always climb up the stairs of the
admin building to get to the map, but he was afraid he might bump around in
the dark and trip over something.  After all, he visited a lot of campuses
at night and they were always different, even the admin buildings, whilst
very similar, were of different sizes.  The maps, however, would be the same
everywhere.


==================================
$$$$$$$$   1999-01-08 addendum   $$$$$$$$
==================================

Let me share my thoughts with you on three possible uses of the enchilada,
with which I am very familiar.

Example 1.  As a backup tool, Hiback does not want to do a DBOPEN of the
database and do a bunch of DBGETs and DBFINDs and DBINFOs.  What Hiback
wants to do is know exactly where to go (whether in the root file or in an
external file/set, it really doesn't matter) to retrieve the HFS file names
of all the files the DBA wants to backup as an IMAGE database entity.  I am
not totally sold on the idea of an external file/set, because I do like the
elegance of having the metadata in the root file.  But I also know that the
root file is never used to store user data.  This would be a change.  This
is an example of why the data should be easily accessible without going
through IMAGE.  Also a user application would probably never want to
retrieve this information, though I am sure somebody could make a case why a
production program would need that information.

Example 2.  Let's say we decide to keep the split/update information in the
enchilada.  We must now find a wait to store this information, at some
point, so that it can be used later.  The program IMAGESQL.PUB.SYS is used
to attach/detach and split & update columns.  Yes, this information is at
the column and table level, not at item or field or dataset level.  A column
in one table can be split one way and the same column in another table can
be split another way, or not at all.  To complicate things, a column (item)
type can be updated at the field level (in one table) and at the item level
(across all tables).  Also be aware that a column which has been created by
splitting a column, can be split again creating new columns, or updated.

Another important thing to capture in an attached database is the user
information.  This is a record for each user that has been granted access to
the database.  We must also record the name of the ATCINFO file.  So, how we
capture this information and how do we make use of this?  One could say,
let's update IMAGESQL.PUB.SYS to record this information, but what about the
existing attached IMAGE databases?  I go back to my earlier post and say
that it would be easier to have a program such as DBTSQL updated to manage
this data for the enchilada.  What I envisage is DBTSQL (which already does
all the decoding of the ATC file and creates a file for IMAGESQL.PUB.SYS to
use) placing its data in the enchilada, in the form of IMAGESQL commands.
This could be done automatically at detach time, or whenever the DBA wants
it updated. One could then simply extract the data directly from the
enchilada and recreate the attach environment, or we could simply have
IMAGESQL read the data and use the commands.  I am trying to minimize or
eliminate the changes required for IMAGESQL.PUB.SYS.  But the point here is
that the information we want to keep has little to do with IMAGE, it has to
do with SQL columns and tables.  So a production program would never need
this information.

Example 3, storing default values.  I have always been a proponent of having
the default value for a field (it should not be at the item level), kept in
the label of the dataset.  But let's go ahead and use the enchilada.  Let's
stipulate that we have the data loaded in the enchilada, according to the
wishes of the DBA.  How is it going to be used?  We all agree the intent of
this one is to have IMAGE automatically store the default value for that
field in a new record added by DBPUT, when the field is not specified.  If
it's not specified in the DBPUT, the program is not concerned with that
field.  Therefore the program would probably not be expanded to encompass a
DBINFO call and subsequent decoding of the returned information to find out
what the value should be.  This means it is up to IMAGE to do the job, DBPUT
to be precise.  So DBPUT will have to be updated to have that processing
added to it.  (Don't get me wrong, I would love the default value, it should
have been in there from the beginning, but I don't think we will be able to
convince HP to modify DBPUT.  Then again, we won't know that until we ask.)
Again, a production program would not use this.

The DBA should be able to add/modify/delete the data and should be able to
list it, but I do not see user programs using this data.  


==================================
$$$$$$$$   1999-02-24 addendum   $$$$$$$$
==================================

Consider the pros of having an extra PM file to house the data.

1- It does not affect the existing database.  We could have one of those
flags that say the DB has an enchilada, a la HWMPUT or TPI.  I believe there
is currently a spare bit in RL'FLAGS'2.  So turning on this bit would either
create a new or use the existing file.  This can be controlled with DBUTIL,
enable/disable <dbname> for ENCHILADA or METADATA.  Alternatively we could
use the ILR flag which should be off anyways.

2- If the database is moved to another group/account, the file could follow
it, or not.  It's up to the backup tools or the DBA to handle this.  Either
way, the file is not fully qualified in the root file, so it's easy to move.

3- If the database is moved to a system which does not know anything about
the enchilada, there is no issue.  The root file would not be impacted, it
will just have a currently unused bit turned on.  The enchilada file is kept
with the database or can be removed independently.

4- If the database is destroyed and recreated, the metadata can survive this
cycle and be used with the new database. 

5- The file is extensible and really has no limit and will not impact the
existing database unlike storing the data in the root file.


The cons of having an extra PM file are as follows:

1- Having to keep track of yet another file.

2- Creating some form of data management to have simple, quick access to the
data.


==================================
$$$$$$$$   1999-10-20 addendum   $$$$$$$$
==================================

SUMMARY:

- I propose to place the enchilada in a privmode file associated with the
data base.

- There may be a flag in the root file indicating the presence of this file.
DBUTIL will be able to enable/disable/report on it.

- A utility would be provided to load and remove entries in this file.  By
default, the utility would load all the files of the database and include TC
and ATC files.

- The file name could be the name of the root file with "EF" appended to it.

- The records in the e-file would be simple:  CODE, TIMESTAMP, PAYLOAD.
===================================