Home > Building a Corporate Testing Lab
Initiative G19 - Establish Enterprise Architecture Lab
Description
Tactical / Strategic
To coordinate EDD
technology investigations and to evaluate new technologies, the BDA
will require a technology investigation and evaluation process.
This BDA process will be responsible for identifying and being informed
of new technologies and evaluating them for incorporation into the BDA.
Input or initiators into this process can come from a number of sources
including: ITDQC, CIT/DIT, ITB Business Consultant’s interaction with
customers, new business proposals, technology vendors and the BDA review
process directly. To support this BDA need, ITB should establish and
maintain an Enterprise Architecture Lab (EAL). As an operational
service, the EAL will also provide a greatly needed resource to support
application acceptance, performance and production testing.
Objectives / Gaps
To establish a central IT resource used for the evaluation and integration of new technologies and new releases of installed base technologies.
To establish an enterprise architecture lab environment where these new technologies can be evaluated and business applications are production readiness tested.
This Initiative addresses, at least in part, the following identified gap(s):
Priority
Low / Medium / High / Very High
Estimated Benefits
Low / Medium / High / Very High
Estimated Risks
Low / Medium / High / Very High 1
Tasks to be Completed
Duration and Deliverables
Duration
<1 month / 1-3months / 3-6 months / 6-12 months / 1-2 year / >2 year / Ongoing
Deliverables
Estimated Costs
Unquantifiable / No Cost / 0-100K / 100-500K / 500K-1M / 1M-5M / 5M-10M / >10M
Project Governance
Recommended project owner(s)
The Business Driven Architecture (BDA) has determined that it takes significant time and labor to orchestrate major, enterprise-wide improvements in the computing environment, such as an operating system upgrade or the implementation of a new messaging system or the deployment of new production systems. These deployments take months of planning and evaluation, piloting and production transition. In addition, the process requires dedicated hardware, software, expert evaluators and management supervision. EDD can reduce the costs, personnel requirements and time to perform major upgrades and Technology Implementations, by establishing a Enterprise Architecture Lab (EAL).
An EAL is chartered to evaluate emerging technologies in support of an organization's business goals. The lab contains the hardware, software, communications and personnel required to evaluate products and services deemed beneficial for corporate deployment. Enterprises save money and time, keep knowledge in-house, improve ties with vendors for the newest versions, get faster technical support, perform beta tests and provide input to product developers to improve their product offerings.
The BDA has or will develop standards for workstations, servers, communications equipment, office software and so on. The components of this list of standard products will be those running in the lab. Any new products, upgrades, fixes or new releases will be evaluated against all the other products on the standard products list to ensure interoperability and backward compatibility. Evaluations can be specific to one application-- such as a Human Resources database search engine--or applicable to the entire organization--such as a Web browser, a digital certificate service or a high-speed color laser printer.
The lab also can be used as an internal resource center, a place where interested users can come to test new hardware and software before recommending its installation or purchase. For example:
Several steps are required to get the lab established and fully functional. ITB needs to fund the construction of the lab; choose the lab personnel; select and procure the hardware, software and communications resources; build processes for conducting evaluations, purchasing and promotion; and manage the lab effectively on a daily basis.
BDA suggests that the sponsor of the lab be the Deputy Director of ITB. Within the executive staff, the sponsor best knows what the lab contains, what will be tested and delivered and what the added value to the enterprise will be. It is necessary that the sponsor of the lab be in the executive staff. The sponsor should be willing to follow the progress of the lab from conception through to day-to-day operation, providing executive insight and support for additional capital expenditures when required and promoting the lab's services to other executives and even to customers. The executive sponsor may wish to have a subordinate take day to day responsibility for the sponsorship requirements, but ultimately the executive sponsor takes the strategic responsibility for the lab
Where should the funding for the EAL come from?
Every branch will or should benefit from the testing that is performed in the lab and from the accumulation of expertise in new and emerging technologies housed there. But there will be a yearly competition with all the other branches vying for corporate resources.
All the hardware and software purchased for that specific project can be redeployed to the lab for more general testing needs. For example, if the enterprise wants to implement electronic software-distribution capabilities for the entire enterprise, funds should be set aside in this budget to procure the requisite hardware and software needed to test and deploy this service.
Other organizations so that employees can use the lab to educate themselves on the latest technology within their particular area of interest. Training generally has hardware, software and space allocated for their purposes and piggy-backing on this may be a good interim step.
The Managers Conference equipment
The funding to implement a full-blown test lab requires a cost-benefit analysis (CBA) that includes tangible and intangible costs.
Staffing the lab will be easier than most technical jobs. Working with the Architecture Team, staff will be required to be familiar with the latest products, conduct extensive market research, attend industry trade shows, keep up with the trade through attending vendor presentations, and reading trade journals.
There will be a need at least three full-time staff to start, a technical manager and two staff. As the workload increases additional full-time employees or part-time people will be needed for the big evaluations, such as selecting a new standard desktop.
At least one lab technician will be needed, who will set up the tests, keep installed releases current (latest OS release and patches, service packs, selected messaging applications and Web browsers) and keep the lab itself neat and tidy.
Finding qualified personnel for a EAL is difficult. Qualifications of staff would be
The lab should be located in a facility with a 10' high ceiling floor space should be sufficient to open the lab and add equipment for two years without being cramped for space or having to move.
The high ceiling is necessary for overhead cable racks that will keep the floor area clean. Existing raised floor area may be used, but this makes cabling changes cumbersome.
Infrastructure needed is:
Offices
Test areas
A space-saving idea is to install master switches that use one monitor, mouse and keyboard to control several machines.
Growth is sure, and it is far easier and less costly to build growth in during the construction phase.
Because the lab personnel will be working for long stretches of time testing products, there is the need for an environment that is soothing and helpful in reducing glare and strain. Use non-glare neon overhead lights in parabolic fixtures. Paint the lab walls a soft pastel color. Consider the ergonomics of the height and angle of the monitors and location of the keyboard and mouse to prevent such physical ailments as carpal tunnel syndrome. Select flooring and racks that are static resistant.
Air conditioning in the area must be adjustable to insure that the heat load of equipment and staff is compensated for.
Secure storage space in the form of floor to ceiling shelves and file cabinets is a must.
Adequate storage space is crucial. During new hardware evaluations, vendors will ship their products in big boxes and these will need to be stored while the test is in progress and then used to ship the products back again at the conclusion of the test. Plus, there will be a need for space sufficient to hold all the product documentation and software packages that are loaded on the test machines.
There will be a need for security both physical and logical security for the lab.
Physical security involves controlling access to the lab room. Typical methods are installing locks, finger locks, access badge readers and so on. Access will be limited to LAB personnel, accompanied personnel and testing groups.
Logical security protects access to the computer equipment and software programs by means of administrator passwords or strong authentication such as public/private key access control, digital signatures and secure cards.
The EAL should mirror the standard Application Development, office and distributed computing production environments in use within the enterprise. The lab will perform client and server testing as well as performance, scalability, functionality and integration testing. The lab should reflect the organization's baseline:
Specific requirements are:
To test the features and functions of distributed client applications, such as workflow
To test forward and backward compatibility for hardware and software upgrades, fixes and patches, for example.
For performance and scalability testing.
As reflected in the enterprise inventory
Sufficiently large enough that allows
for growth in power requirements and in the number of devices
Digital linear technology with significant growth capacity built in.
Running those applications and services that the organization always uses
As defined in the standard desktop platform.
Beneficial Requirements
Give users an opportunity to take their work back with them after configuring a client or server application and are also convenient for multi-boot requirements.
Hardware and software
Telephones:
Equipment should be tagged to the lab for inventory control purposes and to reduce the possibility of it "wandering away." Lab personnel need to keep equipment serial numbers, specifications and configurations for each device on hand for service calls and upgrades.
Software selected to run in the test lab must reflect the company's standard operating systems, network protocol stacks, Web applications, virus control as well as products for office applications, such as e-mail, calendaring, spreadsheet, database, workflow, etc.
Copies of standard software configurations for groups such as secretaries, executives, power users and developers should be running on stable machines.
Any development tool kits and application development programs that are used within the organization. Any custom software that uses the standard software applications must be represented in the lab as well.
There will be a need for current releases of the all the standard software products for both clients and servers as well as the back releases to test interoperability. Getting in new release and new product software for evaluation and feedback is beneficial to both the organization and the vendor but should be discouraged in the early versions.
Servers and clients should be fully populated with standard configuration software
Backup and restore software and hardware for the entire lab is needed. Several software packages are on the market that can automate this process for scheduled nightly backups.
Critical to the success of the test lab is its connectivity with the entire enterprise. The lab needs to be a microcosm of the organization at large in terms of LAN, WAN and Internet connectivity. Network operating systems, network protocols, hubs, routers and switches should model those deployed within the enterprise. The lab needs to have connectivity to mainframes and minicomputers at HWDC as well.
A fast Internet connection is needed so that staff can conduct research on new products, services and technology and to test internet based applications with simulated loads
Also needed is to have modems and wireless connections to mimic remote-access configurations for mobile workers.
Having a test network segment for the lab is necessary so that heavy testing can be accomplished without injecting heavy traffic over the production network. The segment should be routed to the production network and be ready for use when this connection when needed.
The segment should be fire-walled so that the test network and the production network do not interfere with each other.
Scheduling involves determining who gets to use the services, for how long and during what hours. Determining who gets priority is no easy task. There will always be more activities than resources to do them.
Enterprise Architecture Priorities
Procedures
Requests to use the lab must be funneled through the lab manager or representative of the lab's executive champion. This procedure will ensure that all requests are treated as equitably as possible and that the lab's resources are not over-taxed. It's important to have agreement on the request procedures from those who will use the lab most often. They need to feel that the lab is accessible, ready, willing and able to meet their testing needs.
The schedule for the lab will be posted on the lab's Intranet Web site, and the front door of the lab. The schedule will reflect what software will be loaded on the machines, what hardware is planned for testing, what beta code is in, what major evaluations are in progress and their status.
Test sponsors may provide/require the technology for their particular test. If a product is to become pare ot the standard suit, the product is procured for the lab as a part of the normal standard procurement process. Conversely, if a product is required that is specific to a particular test, that product should be procured by the project sponsor.
Identify a buyer who works with the lab. Staff will be coming to them on a regular basis to arrange for evaluation software and hardware to be delivered as quickly as possible to the lab or to any distributed testing locations.
Establish master evaluation agreements with the organization's standard products vendors. These agreements must be formalized and detail the requirements for both the vendor and the lab. There must be a suitable standard agreement between your company and the major vendors.
Have product testing agreements in place with your major vendors so that the new code and hardware will automatically be shipped to you as soon as it becomes available.
The lab should maintain copies of all agreements for reference
Vendors play by certain rules. They'll ship the product--overnight if needed that badly. They'll give a defined evaluation period such as 60 to 120 days depending on the complexity of the product and potentially how big the sale of their product might be within the enterprise. They'll offer support and maybe even on-site installation. They'll help with competitive analyses and technical specifications and provide case studies and customer references. And at the end of the evaluation period, they'll fully expect the organization to buy the product or return it to them in its original condition and original packaging material.
Always get a direct technical support contact from the vendor for your evaluation period as appropriate. This person/group is invaluable in installing and configuring the hardware/software for testing. Going through the normal channel of the 800 number call to be placed on hold for customer service just won't meet the needs of the organization. Also get access to the product developer and marketing people so the organization can tell strategically where the product is going and what is planned for upcoming releases.
Vendor personnel should not have access to the Lab except as an escorted guest.
Lab activities must be communicated to the entire enterprise. This internal marketing and information transfer of lessons learned, configuration advice and knowledge on various new products, services, upgrades, represents a value-added service to corporation and can reap the benefits of continuous funding and resource allocation.
Launch the lab in a big way with a ribbon-cutting ceremony and coverage on the company's Web site, and newsletter. Status reports for the lab can be pushed to interested parties from a listserver placed on the Web site.
When the lab offers the computing, consulting and evaluation services to others within the corporation, the lab performs a valuable service. Constantly seek out opportunities to help groups within the organization with their problems, keeping. current on real world problems within the organization. Set up a showcase to display all the standard products and services and emerging technology that the lab has to offer.
Detail the lab’s evaluation plans and accomplishments on the lab's Intranet home page. A example would be, a chart showing the lab's usage (number of products tested, number of users testing, resources utilization, etc.) on a monthly basis will help to manage user expectations and convey to management the extent of the lab's use within the enterprise.
From time to time the test lab will become involved or spearhead large-scale evaluations for items such as a new standard desktop or a new enterprise wide calendar/scheduler system. There will be a need to go through several steps to conduct a thorough product evaluation that will stand up under scrutiny by both architecture, management and end users alike. Architecture or management will use the evaluation results as technical, cost and risk input into their decision making process. (Smaller evaluations may not require all of the steps outlined below.)
The steps for lab evaluations are:
Here a large committee meets over several occasions to develop a requirements definition. Members from the test lab must be present in these meetings or lead them. Expert testers and customer service personnel also must be present to assist in formulating the requirements. The most important characteristics of a requirement are that they must be testable, quantifiable and measurable. A requirement of "has an intuitive user interface" cannot be tested, but a requirement of "takes only two clicks to book an appointment" is quantifiable.
Requirements are separated into musts and wants. A must is a hard requirement--one that, if missing, will throw the product out. A typical “must” would be "supports Windows NT clients." A want is something that is not required but is highly desirable. A typical want would be "context-sensitive help." Wants are weighted on a scale, usually from 1 to 5 with the most important features receiving a 5. The features rating 1s you could live without. The products are then scored on a scale of 1 to 5 on how well they meet the want requirement. Multiplying the weights by the score given to the item during testing leads to a total weighted score for the product. The totals are then examined and the top three or so products are brought in for a pilot program.
These experts are those proficient with a particular software, hardware or communications product or application. These are the power users or heavy users in the organization that are brought in for major tests like an operating system upgrade.
This can be as simple as an interoperability check-list of all the standard products and how the new product performs with them. Or, the test plan can be very detailed and consider all the ramifications of implementing the product with in the corporate computing environment. Scalability, performance testing, impact on the network, security, end user functionality, product administration, maintenance and so on would be included in a formal test plan.
Timing is crucial here so that everything is ready to be tested when the testers are ready to test. Up-front planning and coordination will save time and aggravation in the long run. There will be a need to have an overall test coordinator to orchestrate the testing, troubleshoot the systems and be the focal point for contacting the vendor for technical support.
The pilot is generally performed in an influential division where some of the testing experts reside. The pilot gives the enterprise an opportunity to work more closely with the vendor's installation and support staff and a chance to fully understand how the product will function within the production environment of the enterprise.
A product rollout plan and statement
of work expected of the vendor will be created, contracts will be signed
and delivery of the new equipment or software will begin. At this point,
the test lab, customer support and the expert testers will be involved
in planning for transitioning the product into production.
Reference: White Paper
2 Based, in part, on a document located at Network Computing Online – Network Design Manual -http://www.networkcomputing.com/netdesign/series.htm
All Rights Reserved Powered by Free Document Search and Download
Copyright © 2011